[GitHub] incubator-hawq issue #1333: HAWQ-1582. hawq ssh cmd bug when pipe in cmd
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/1333 +1 ---
[GitHub] incubator-hawq pull request #1286: HAWQ-1525. Segmentation fault occurs if r...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1286 HAWQ-1525. Segmentation fault occurs if reindex database when loading⦠⦠data from Hive to HAWQ using hcatalog You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1525 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1286.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1286 commit 3f272db61ae70811c9a035a4fd8594cbb69200eb Author: wcl14 Date: 2017-09-12T08:59:02Z HAWQ-1525. Segmentation fault occurs if reindex database when loading data from Hive to HAWQ using hcatalog ---
[GitHub] incubator-hawq pull request #1265: HAWQ-1500. HAWQ-1501. HAWQ-1502. Support ...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1265#discussion_r126856563
--- Diff: depends/libhdfs3/src/client/CryptoCodec.cpp ---
@@ -0,0 +1,163 @@
+/
+ * 2014 -
+ * open source under Apache License Version 2.0
+ /
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "CryptoCodec.h"
+#include "Logger.h"
+
+using namespace Hdfs::Internal;
+
+namespace Hdfs {
+
+/**
+ * Construct a CryptoCodec instance.
+ * @param encryptionInfo the encryption info of file.
+ * @param kcp a KmsClientProvider instance to get key from kms server.
+ * @param bufSize crypto buffer size.
+ */
+CryptoCodec::CryptoCodec(FileEncryptionInfo *encryptionInfo,
std::shared_ptr kcp, int32_t bufSize) :
encryptionInfo(encryptionInfo), kcp(kcp), bufSize(bufSize)
+{
+
+ /* Init global status. */
+ ERR_load_crypto_strings();
+ OpenSSL_add_all_algorithms();
+ OPENSSL_config(NULL);
+
+ /* Create cipher context. */
+ encryptCtx = EVP_CIPHER_CTX_new();
+ cipher = NULL;
+
+}
+
+/**
+ * Destroy a CryptoCodec instance.
+ */
+CryptoCodec::~CryptoCodec()
+{
+ if (encryptCtx)
+ EVP_CIPHER_CTX_free(encryptCtx);
+}
+
+/**
+ * Get decrypted key from kms.
+ */
+std::string CryptoCodec::getDecryptedKeyFromKms()
+{
+ ptree map = kcp->decryptEncryptedKey(*encryptionInfo);
+ std::string key = map.get("material");
+
+ int rem = key.length() % 4;
+ if (rem) {
+ rem = 4 - rem;
+ while (rem != 0) {
+ key = key + "=";
+ rem--;
+ }
+ }
+
+ std::replace(key.begin(), key.end(), '-', '+');
+ std::replace(key.begin(), key.end(), '_', '/');
+
+ LOG(INFO, "material is :%s", key.c_str());
+
+ key = KmsClientProvider::base64Decode(key);
+ return key;
+
+
+}
+
+/**
+ * Common encode/decode buffer method.
+ * @param buffer the buffer to be encode/decode.
+ * @param size the size of buffer.
+ * @param enc true is for encode, false is for decode.
+ * @return return the encode/decode buffer.
+ */
+std::string CryptoCodec::endecInternal(const char * buffer, int64_t size,
bool enc)
+{
+ std::string key = encryptionInfo->getKey();
+ std::string iv = encryptionInfo->getIv();
+ LOG(INFO, "CryptoCodec : endecInternal info. key:%s iv:%s buffer:%s
size:%ld is_encode:%b", key.c_str(), iv.c_str(), buffer, size, enc);
+
+ /* Get decrypted key from KMS */
+ key = getDecryptedKeyFromKms();
+
+ /* Select cipher method based on the key length. */
+ if (key.length() == KEY_LENGTH_256) {
+ cipher = EVP_aes_256_ctr();
+ } else if (key.length() == KEY_LENGTH_128) {
+ cipher = EVP_aes_128_ctr();
+ } else {
+ cipher = EVP_aes_192_ctr();
+ }
--- End diff --
If key.length() is out of range, should we throw an expectation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1265: HAWQ-1500. HAWQ-1501. HAWQ-1502. Support ...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1265#discussion_r126858617
--- Diff: depends/libhdfs3/src/client/CryptoCodec.cpp ---
@@ -0,0 +1,163 @@
+/
+ * 2014 -
+ * open source under Apache License Version 2.0
+ /
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "CryptoCodec.h"
+#include "Logger.h"
+
+using namespace Hdfs::Internal;
+
+namespace Hdfs {
+
+/**
+ * Construct a CryptoCodec instance.
+ * @param encryptionInfo the encryption info of file.
+ * @param kcp a KmsClientProvider instance to get key from kms server.
+ * @param bufSize crypto buffer size.
+ */
+CryptoCodec::CryptoCodec(FileEncryptionInfo *encryptionInfo,
std::shared_ptr kcp, int32_t bufSize) :
encryptionInfo(encryptionInfo), kcp(kcp), bufSize(bufSize)
+{
+
+ /* Init global status. */
+ ERR_load_crypto_strings();
+ OpenSSL_add_all_algorithms();
+ OPENSSL_config(NULL);
+
+ /* Create cipher context. */
+ encryptCtx = EVP_CIPHER_CTX_new();
+ cipher = NULL;
+
+}
+
+/**
+ * Destroy a CryptoCodec instance.
+ */
+CryptoCodec::~CryptoCodec()
+{
+ if (encryptCtx)
+ EVP_CIPHER_CTX_free(encryptCtx);
+}
+
+/**
+ * Get decrypted key from kms.
+ */
+std::string CryptoCodec::getDecryptedKeyFromKms()
+{
+ ptree map = kcp->decryptEncryptedKey(*encryptionInfo);
+ std::string key = map.get("material");
+
+ int rem = key.length() % 4;
+ if (rem) {
+ rem = 4 - rem;
+ while (rem != 0) {
+ key = key + "=";
+ rem--;
+ }
+ }
--- End diff --
Why still align key as 4 bytes? Should it be encoded as based 64?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1265: HAWQ-1500. HAWQ-1501. HAWQ-1502. Support ...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1265#discussion_r126856385
--- Diff: depends/libhdfs3/src/client/CryptoCodec.cpp ---
@@ -0,0 +1,163 @@
+/
+ * 2014 -
+ * open source under Apache License Version 2.0
+ /
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "CryptoCodec.h"
+#include "Logger.h"
+
+using namespace Hdfs::Internal;
+
+namespace Hdfs {
+
+/**
+ * Construct a CryptoCodec instance.
+ * @param encryptionInfo the encryption info of file.
+ * @param kcp a KmsClientProvider instance to get key from kms server.
+ * @param bufSize crypto buffer size.
+ */
+CryptoCodec::CryptoCodec(FileEncryptionInfo *encryptionInfo,
std::shared_ptr kcp, int32_t bufSize) :
encryptionInfo(encryptionInfo), kcp(kcp), bufSize(bufSize)
+{
+
+ /* Init global status. */
+ ERR_load_crypto_strings();
+ OpenSSL_add_all_algorithms();
+ OPENSSL_config(NULL);
+
+ /* Create cipher context. */
+ encryptCtx = EVP_CIPHER_CTX_new();
+ cipher = NULL;
+
+}
+
+/**
+ * Destroy a CryptoCodec instance.
+ */
+CryptoCodec::~CryptoCodec()
+{
+ if (encryptCtx)
+ EVP_CIPHER_CTX_free(encryptCtx);
+}
+
+/**
+ * Get decrypted key from kms.
+ */
+std::string CryptoCodec::getDecryptedKeyFromKms()
+{
+ ptree map = kcp->decryptEncryptedKey(*encryptionInfo);
+ std::string key = map.get("material");
+
+ int rem = key.length() % 4;
+ if (rem) {
+ rem = 4 - rem;
+ while (rem != 0) {
+ key = key + "=";
+ rem--;
+ }
+ }
+
+ std::replace(key.begin(), key.end(), '-', '+');
+ std::replace(key.begin(), key.end(), '_', '/');
--- End diff --
Can you give the comment for why replacing them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1265: HAWQ-1500. HAWQ-1501. HAWQ-1502. Support ...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1265#discussion_r126857880
--- Diff: depends/libhdfs3/src/client/HttpClient.cpp ---
@@ -0,0 +1,337 @@
+/
+ * 2014 -
+ * open source under Apache License Version 2.0
+ /
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "HttpClient.h"
+#include "Logger.h"
+
+using namespace Hdfs::Internal;
+
+namespace Hdfs {
+
+#define CURL_SETOPT(handle, option, optarg, fmt, ...) \
+res = curl_easy_setopt(handle, option, optarg); \
+if (res != CURLE_OK) { \
+THROW(HdfsIOException, fmt, ##__VA_ARGS__); \
+}
+
+#define CURL_SETOPT_ERROR1(handle, option, optarg, fmt) \
+CURL_SETOPT(handle, option, optarg, fmt, curl_easy_strerror(res));
+
+#define CURL_SETOPT_ERROR2(handle, option, optarg, fmt) \
+CURL_SETOPT(handle, option, optarg, fmt, curl_easy_strerror(res), \
+errorString().c_str())
+
+#define CURL_PERFORM(handle, fmt) \
+res = curl_easy_perform(handle); \
+if (res != CURLE_OK) { \
+THROW(HdfsIOException, fmt, curl_easy_strerror(res),
errorString().c_str()); \
+}
+
+
+#define CURL_GETOPT_ERROR2(handle, option, optarg, fmt) \
+res = curl_easy_getinfo(handle, option, optarg); \
+if (res != CURLE_OK) { \
+THROW(HdfsIOException, fmt, curl_easy_strerror(res),
errorString().c_str()); \
+}
+
+#define CURL_GET_RESPONSE(handle, code, fmt) \
+CURL_GETOPT_ERROR2(handle, CURLINFO_RESPONSE_CODE, code, fmt);
+
+HttpClient::HttpClient() : curl(NULL), list(NULL) {
+
+}
+
+/**
+ * Construct a HttpClient instance.
+ * @param url a url which is the address to send the request to the
corresponding http server.
+ */
+HttpClient::HttpClient(const std::string &url) {
+ curl = NULL;
+ list = NULL;
+ this->url = url;
+}
+
+/**
+ * Destroy a HttpClient instance.
+ */
+HttpClient::~HttpClient()
+{
+ destroy();
+}
+
+/**
+ * Receive error string from curl.
+ */
+std::string HttpClient::errorString() {
+ if (strlen(errbuf) == 0)
+ return "";
+ return errbuf;
+}
+
+/**
+ * Curl call back function to receive the reponse messages.
+ * @return return the size of reponse messages.
+ */
+size_t HttpClient::CurlWriteMemoryCallback(void *contents, size_t size,
size_t nmemb, void *userp)
+{
+ size_t realsize = size * nmemb;
+ if (userp == NULL || contents == NULL) {
+ return 0;
+ }
+ ((std::string *)userp)->append((const char *)contents, realsize);
+ LOG(DEBUG2, "HttpClient : Http response is : %s", ((std::string
*)userp)->c_str());
+ return realsize;
+}
+
+/**
+ * Init curl handler and set curl options.
+ */
+void HttpClient::init() {
+ if (!initialized)
+ {
+ initialized = true;
+ if (curl_global_init(CURL_GLOBAL_ALL)) {
+ THROW(HdfsIOException, "Cannot initialize curl client
for KMS");
+ }
+ }
+
+ curl = curl_easy_init();
+ if (!curl) {
+ THROW(HdfsIOException, "Cannot initialize curl handle for KMS");
+ }
+
+CURL_SETOPT_ERROR1(curl, CURLOPT_ERRORBUFFER, errbuf,
+"Cannot initialize curl error buffer for KMS: %s");
+
+errbuf[0] = 0;
+
+CURL_SETOPT_ERROR2(curl, CURLOPT_NOPROGRESS, 1,
+"Cannot initialize no progress in HttpClient: %s: %s");
+
+CURL_SETOPT_ERROR2(curl, CURLOPT_VERBOSE, 0,
+"Cannot initialize no verbose in HttpClient: %s: %s
[GitHub] incubator-hawq pull request #1213: HAWQ-1426. Fix bug that hawq extract meet...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1213 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1213: HAWQ-1426. Fix bug that hawq extract meets error...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/1213 @ictmalili @zhangh43 @interma please review, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1213: HAWQ-1426. Fix bug that hawq extract meet...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1213 HAWQ-1426. Fix bug that hawq extract meets error after the table was ⦠â¦reorganized You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1426 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1213.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1213 commit 73a73f717b82a64371a7dff0d3051c65b4652588 Author: Chunling Wang Date: 2017-04-07T08:28:22Z HAWQ-1426. Fix bug that hawq extract meets error after the table was reorganized --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1201: HAWQ-1418. Move print executing command a...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1201 HAWQ-1418. Move print executing command after setup logging You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1418 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1201.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1201 commit 1e9ea99458a251c3084e69147bdde2c19e702049 Author: Chunling Wang Date: 2017-03-28T10:15:00Z HAWQ-1418. Move print executing command after setup logging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1200: HAWQ-1140. Rename yml file name and table...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1200 HAWQ-1140. Rename yml file name and table name in TestUsage2Case1Erro⦠â¦rHashTableRegistry You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1140 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1200.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1200 commit 8ab2375a6a1a3bd7a31823d56eb1e71219deb49e Author: Chunling Wang Date: 2017-03-28T09:31:55Z HAWQ-1140. Rename yml file name and table name in TestUsage2Case1ErrorHashTableRegistry --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1199: HAWQ-1418. Print executing command for ha...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1199 HAWQ-1418. Print executing command for hawq register You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1418 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1199.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1199 commit 2187dcefc4ad39dbd6ea19da3f6fb3e00b623b33 Author: Chunling Wang Date: 2017-03-28T09:08:35Z HAWQ-1418. Print executing command for hawq register --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1161: HAWQ-1377. Add more information for Range...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1161 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1161: HAWQ-1377. Add more information for Ranger relat...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/1161 @paul-guo- Thanks for review. I will modify with your comments. But I don't think we should merge three GUCs into one, it follows HAWQ style. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1161: HAWQ-1377. Add more information for Range...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1161 HAWQ-1377. Add more information for Ranger related GUCs in default ha⦠â¦wq-site.xml. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1377 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1161.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1161 commit 1228eb8f311c8b73264c500ad0fa6691d0bf2296 Author: Chunling Wang Date: 2017-03-06T04:10:33Z HAWQ-1377. Add more information for Ranger related GUCs in default hawq-site.xml. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1158: HAWQ-1359. Fix bugs in policy tests for H...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1158 HAWQ-1359. Fix bugs in policy tests for HAWQ with Ranger enabled and ⦠â¦add test for HAWQ-1367 You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1359 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1158 commit 68013fac13a757489d7a7ebe361065826ea5d66a Author: Chunling Wang Date: 2017-03-03T03:23:32Z HAWQ-1359. Fix bugs in policy tests for HAWQ with Ranger enabled and add test for HAWQ-1367 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1154: HAWQ-1367. HAWQ can access to user tables...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1154 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1154: HAWQ-1367. HAWQ can access to user tables that h...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/1154 @stanlyxiang @zhangh43 @ictmalili Please review, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1154: HAWQ-1367. HAWQ can access to user tables...
GitHub user wcl14 reopened a pull request: https://github.com/apache/incubator-hawq/pull/1154 HAWQ-1367. HAWQ can access to user tables that have no permission wit⦠â¦h fallback check table. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1367 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1154 commit 2536993c42c5c3673c7fb21ef72660cb51f2b8bf Author: Chunling Wang Date: 2017-02-28T10:08:59Z HAWQ-1367. HAWQ can access to user tables that have no permission with fallback check table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1154: HAWQ-1367. HAWQ can access to user tables...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1154 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1154: HAWQ-1367. HAWQ can access to user tables...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1154 HAWQ-1367. HAWQ can access to user tables that have no permission wit⦠â¦h fallback check table. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1367 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1154 commit 2536993c42c5c3673c7fb21ef72660cb51f2b8bf Author: Chunling Wang Date: 2017-02-28T10:08:59Z HAWQ-1367. HAWQ can access to user tables that have no permission with fallback check table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1152: Hawq 1365. Print out detailed schema info...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1152#discussion_r103395505
--- Diff: src/backend/catalog/namespace.c ---
@@ -1982,6 +1982,13 @@ recomputeNamespacePath(void)
elog(DEBUG3, "recompute search_path[%s] when acl_type
is ranger", namespace_search_path);
}
}
+ else
+ {
+ if (aclType == HAWQ_ACL_RANGER && debug_query_string != NULL)
+ {
+ last_query_sign = string_hash(debug_query_string,
strlen(debug_query_string));
+ }
+ }
--- End diff --
How about set current_qurey_sign before the if statement 'if
(namespaceSearchPathValid && namespaceUser == roleid)' and set last_query_sign
after the if statement? Can this make the return condition more clear?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1088: HAWQ-1276. The error message is not frien...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1088#discussion_r96180237
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -366,7 +365,7 @@ static size_t write_callback(char *contents, size_t
size, size_t nitems,
if (curl->response.buffer == NULL)
{
/* out of memory! */
- elog(WARNING, "not enough memory for Ranger response");
+ elog(WARNING, "cannot allocate memory for ranger response");
--- End diff --
So should the comment be consistent with the code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1088: HAWQ-1276. The error message is not frien...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1088#discussion_r96178079
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -366,7 +365,7 @@ static size_t write_callback(char *contents, size_t
size, size_t nitems,
if (curl->response.buffer == NULL)
{
/* out of memory! */
- elog(WARNING, "not enough memory for Ranger response");
+ elog(WARNING, "cannot allocate memory for ranger response");
--- End diff --
Why do you modify the out of memory error message. I think the original one
is more specified, and may be better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1074: HAWQ-1249. Don't do ACL checks on segment...
Github user wcl14 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1074#discussion_r95105180 --- Diff: src/backend/executor/execMain.c --- @@ -1912,45 +1912,10 @@ InitPlan(QueryDesc *queryDesc, int eflags) * rangetable here --- subplan RTEs will be checked during * ExecInitSubPlan(). */ - if (operation != CMD_SELECT || - (Gp_role != GP_ROLE_EXECUTE && -!(shouldDispatch && cdbpathlocus_querysegmentcatalogs))) --- End diff -- I think it will not introduce regression on master, but I haven't completed all feature tests yet. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1074: HAWQ-1249. Don't do ACL checks on segment...
Github user wcl14 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1074#discussion_r95103133 --- Diff: src/backend/catalog/aclchk.c --- @@ -3006,8 +3007,8 @@ pg_database_aclmask(Oid db_oid, Oid roleid, Oid ownerId; cqContext *pcqCtx; - /* Superusers or on QE bypass all permission checking. */ - if (GP_ROLE_EXECUTE == Gp_role || superuser_arg(roleid)) --- End diff -- Ok, I will change these, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1074: HAWQ-1249. Don't do ACL checks on segment...
Github user wcl14 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1074#discussion_r95103095 --- Diff: src/backend/executor/execMain.c --- @@ -1912,45 +1912,10 @@ InitPlan(QueryDesc *queryDesc, int eflags) * rangetable here --- subplan RTEs will be checked during * ExecInitSubPlan(). */ - if (operation != CMD_SELECT || - (Gp_role != GP_ROLE_EXECUTE && -!(shouldDispatch && cdbpathlocus_querysegmentcatalogs))) --- End diff -- I have debugged these codes, and am sure they could be called on segments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1074: HAWQ-1249. Don't do ACL checks on segments
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/1074 @zhangh43 @linwen @stanlyxiang Please review, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1074: HAWQ-1249. Don't do ACL checks on segment...
Github user wcl14 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1074#discussion_r95102766 --- Diff: src/backend/catalog/aclchk.c --- @@ -224,16 +224,17 @@ restrict_and_check_grant(bool is_grant, AclMode avail_goptions, bool all_privs, * If we found no grant options, consider whether to issue a hard error. * Per spec, having any privilege at all on the object will get you by * here. +* Superusers or on QE bypass all permission checking. --- End diff -- For ranger, we should still do checks but for normal ACL checks, we should bypass all permission checking for superusers. I should change the comment, and thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1074: HAWQ-1249. Don't do ACL checks on segment...
Github user wcl14 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1074#discussion_r95102563 --- Diff: src/backend/executor/execMain.c --- @@ -1912,45 +1912,10 @@ InitPlan(QueryDesc *queryDesc, int eflags) * rangetable here --- subplan RTEs will be checked during * ExecInitSubPlan(). */ - if (operation != CMD_SELECT || - (Gp_role != GP_ROLE_EXECUTE && -!(shouldDispatch && cdbpathlocus_querysegmentcatalogs))) --- End diff -- Yes, we think these parts of code should be removed for there is no need to do ACL checks on segments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1074: HAWQ-1249. Don't do ACL checks on segment...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1074 HAWQ-1249. Don't do ACL checks on segments You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1249 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1074.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1074 commit cb4c6ca871d46d859b82531b7fd8541ca9cd4693 Author: Chunling Wang Date: 2017-01-05T10:49:59Z HAWQ-1249. Don't do ACL checks on segments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1066: HAWQ-1237. Modify hard code 'select' priv...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1066 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1067: HAWQ-1239. Fail to call pg_rangercheck_ba...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1067 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1067: HAWQ-1239. Fail to call pg_rangercheck_batch() w...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/1067 cc @zhangh43 @stanlyxiang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1067: HAWQ-1239. Fail to call pg_rangercheck_ba...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1067 HAWQ-1239. Fail to call pg_rangercheck_batch() when when 'rte->rtekin⦠â¦d != RTE_RELATION' or 'requiredPerms == 0' You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1239 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1067.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1067 commit 3bdbad45e706b11dfc4dfcbeff1691a35b3f1297 Author: Chunling Wang Date: 2016-12-27T09:52:56Z HAWQ-1239. Fail to call pg_rangercheck_batch() when when 'rte->rtekind != RTE_RELATION' or 'requiredPerms == 0' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1066: HAWQ-1237. Modify hard code 'select' privilege i...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/1066 @zhangh43 @stanlyxiang Please review, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1066: HAWQ-1237. Modify hard code 'select' priv...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1066 HAWQ-1237. Modify hard code 'select' privilege in create_ranger_reque⦠â¦st_json_batch() in rangerrest.c You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1237 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1066.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1066 commit 5934be6f2f665d5732f9feafabc515a647474e75 Author: Chunling Wang Date: 2016-12-27T02:11:13Z HAWQ-1237. Modify hard code 'select' privilege in create_ranger_request_json_batch() in rangerrest.c --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1050: HAWQ-1221. hawq register should error out...
Github user wcl14 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1050#discussion_r92545931 --- Diff: tools/bin/hawqregister --- @@ -1145,7 +1153,7 @@ def main(options, args): failure_handler = FailureHandler(conn) # register -if options.yml_config and ispartition(options.yml_config): +if check_file_exist(options.yml_config) and ispartition(options.yml_config): --- End diff -- I just think check_file_exist() should not be called in the if statement. In this statement, it may be confused the condition for whether is partition or not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1050: HAWQ-1221. hawq register should error out...
Github user wcl14 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1050#discussion_r92410246 --- Diff: tools/bin/hawqregister --- @@ -1145,7 +1153,7 @@ def main(options, args): failure_handler = FailureHandler(conn) # register -if options.yml_config and ispartition(options.yml_config): +if check_file_exist(options.yml_config) and ispartition(options.yml_config): --- End diff -- Should function check_file_exist(options.yml_config) be called in both partition table and non-partition table? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1048: HAWQ-1140. Parallelize test cases for haw...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1048 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1048: HAWQ-1140. Parallelize test cases for haw...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1048 HAWQ-1140. Parallelize test cases for hawqregister usage2. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1140 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1048.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1048 commit 8da85b198f70df5edbecb1cf1e4a780f65f4b357 Author: Chunling Wang Date: 2016-12-12T08:56:58Z HAWQ-1140. Parallelize test cases for hawqregister usage2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1017: HAWQ-1145. Add UDF gp_relfile_insert_for_...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1017 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1017: HAWQ-1145. Add UDF gp_relfile_insert_for_...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1017 HAWQ-1145. Add UDF gp_relfile_insert_for_register and add insert meta⦠â¦data into gp_relfile_node and gp_persistent_relfile_node for HAWQ register You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1145 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1017.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1017 commit ce79777b1c07b28b55a530790adae4bac4dead3e Author: Chunling Wang Date: 2016-11-18T05:07:33Z HAWQ-1145. Add UDF gp_relfile_insert_for_register and add insert metadata into gp_relfile_node and gp_persistent_relfile_node for HAWQ register --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #955: HAWQ-1035. Support partition table registe...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/955 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #986: HAWQ-1034. Delete code for hawq register r...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/986 HAWQ-1034. Delete code for hawq register repair mode You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1034 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/986.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #986 commit 778ab999a95c36999430fea6ea29e7839106d9d1 Author: Chunling Wang Date: 2016-10-31T05:50:14Z HAWQ-1034. Delete code for hawq register repair mode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #955: HAWQ-1035. Support partition table register.
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/955 @linwen @zhangh43 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #971: HAWQ-1113. fix files_same_path/sizes_same_path/se...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/971 @xunzhang Yes, I will do this modification in #955. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #970: HAWQ-1113. Fix bug when files in yaml is d...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/970 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #970: HAWQ-1113. Fix bug when files in yaml is disorder...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/970 @xunzhang Yes, I have enabled the feature test cases for this modification. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #970: HAWQ-1113. Fix bug when files in yaml is d...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/970 HAWQ-1113. Fix bug when files in yaml is disordered, hawq register er⦠â¦ror in force mode. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1113 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/970.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #970 commit 12d40464d6d5f14db769640630cddeea44d5eda0 Author: Chunling Wang Date: 2016-10-21T05:36:32Z HAWQ-1113. Fix bug when files in yaml is disordered, hawq register error in force mode. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #967: HAWQ-1044. Enhance hawq register test case...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/967 HAWQ-1044. Enhance hawq register test cases for normal mode and force⦠⦠mode You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1044 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/967.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #967 commit 3bcc557917cf8962d9963f59d634877f9c262d76 Author: Chunling Wang Date: 2016-10-19T10:35:09Z HAWQ-1044. Enhance hawq register test cases for normal mode and force mode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #955: HAWQ-1035. Support partition table registe...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/955 HAWQ-1035. Support partition table register. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1035 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/955.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #955 commit 716c42949e17736c523d8f3c2462f56817146e01 Author: Chunling Wang Date: 2016-10-10T13:48:10Z HAWQ-1035. Support partition table register. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #950: HAWQ-991. Refator HAWQ Register code to ma...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/950 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #950: HAWQ-991. Refator HAWQ Register code for p...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/950 HAWQ-991. Refator HAWQ Register code for partition table. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-991 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/950.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #950 commit 4f9222792ef4dc0e9c9d130752faaa9e178aa8a0 Author: Chunling Wang Date: 2016-10-01T14:06:51Z HAWQ-991. Refator HAWQ Register code for partition table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #923: HAWQ-1044. Add some error path test cases ...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/923 HAWQ-1044. Add some error path test cases for hawq register (TestUsag⦠â¦e2Case1ErrorEncoding, TestUsage2Case1Bucket0, TestUsage2Case1IncludeDirectory) You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1044 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/923.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #923 commit e5ab81979b7804d0f0946fbb87c13b0add7584d8 Author: Chunling Wang Date: 2016-09-21T02:30:47Z HAWQ-1044. Add some error path test cases for hawq register (TestUsage2Case1ErrorEncoding, TestUsage2Case1Bucket0, TestUsage2Case1IncludeDirectory) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #912: HAWQ-1044. Remove duplicated testcases for...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/912 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #912: HAWQ-1044. Remove duplicated testcases for hawq r...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/912 cc @lili @wengyanqing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #912: HAWQ-1044. Remove duplicated testcases for...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/912 HAWQ-1044. Remove duplicated testcases for hawq resgister usage2 You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1044 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/912.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #912 commit 1be7e8bb2464ea1b4a646db26818668d499428fc Author: Chunling Wang Date: 2016-09-19T09:42:36Z HAWQ-1044. Remove duplicated testcases for hawq resgister usage2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #909: HAWQ-1044. Add normal path testcase for ha...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/909 HAWQ-1044. Add normal path testcase for hawq register when file exist⦠â¦s, file does not exist and --force You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1044 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/909.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #909 commit f87c234e127315aa2a0ab00a0dfe9c3acdef530a Author: Chunling Wang Date: 2016-09-19T03:15:38Z HAWQ-1044. Add normal path testcase for hawq register when file exists, file does not exist and --force --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #900: HAWQ-1033. Combine update and insert into one tra...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/900 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #885: HAWQ-1033. Add --force option for hawq register.
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/885 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r78490699
--- Diff: src/test/feature/ManagementTool/test_hawq_register_usage1.cpp ---
@@ -0,0 +1,128 @@
+#include
+#include
+
+#include "gtest/gtest.h"
+#include "lib/command.h"
+#include "lib/sql_util.h"
+#include "lib/string_util.h"
+#include "lib/hdfs_config.h"
+#include "test_hawq_register.h"
+
+using std::vector;
+using std::string;
+using hawq::test::SQLUtility;
+using hawq::test::Command;
+using hawq::test::HdfsConfig;
+
+TEST_F(TestHawqRegister, TestUsage1ExpectSuccess) {
+ // Register file/folder into HAWQ by specific file/folder name
+
+ SQLUtility util;
+ string rootPath(util.getTestRootPath());
+ string filePath = rootPath + "/ManagementTool/data/parquet200/dat.paq";
+ string folderPath = rootPath + "/ManagementTool/data/parquet200sum/";
+
+ vector ddl_orientation_matrix = {"parquet"};
+ vector distribution_policy_matrix = {"", "DISTRIBUTED RANDOMLY"};
+
+ for(int i = 0; i < ddl_orientation_matrix.size() *
distribution_policy_matrix.size() * 4; ++i) {
+util.execute(hawq::test::stringFormat("drop table if exists t_%s;",
std::to_string(i).c_str()));
+ }
+ auto register_lambda = [&] () {
+int suffix = 0;
+// hawq register -d hawq_feature_test -f
hdfs://localhost:8020/usage1dat.paq t_#
+for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+auto cmd = hawq::test::stringFormat("hdfs dfs -put -f %s
%s/usage1dat.paq", filePath.c_str(), getHdfsLocation().c_str());
+EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with
(appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(),
ddl.c_str(), policy.c_str());
+util.execute(sql); util.query(hawq::test::stringFormat("SELECT *
from t_%s", std::to_string(suffix).c_str()), 0);
+
+cmd = hawq::test::stringFormat("hawq register -d %s -f
%s/usage1dat.paq t_%s", HAWQ_DB, getHdfsLocation().c_str(),
std::to_string(suffix).c_str());
+EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+util.query(hawq::test::stringFormat("select * from t_%s;",
std::to_string(suffix).c_str()), 200);
+util.execute(hawq::test::stringFormat("insert into t_%s
values(201);", std::to_string(suffix).c_str()));
+util.query(hawq::test::stringFormat("select * from t_%s;",
std::to_string(suffix).c_str()), 201);
+
+suffix ++;
+ }
+}
+
+// hawq register -d hawq_feature_test -f
hdfs://localhost:8020/usage1dat.paq -e eof t_#
+for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+auto cmd = hawq::test::stringFormat("hdfs dfs -put -f %s
%s/usage1dat.paq", filePath.c_str(), getHdfsLocation().c_str());
+EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with
(appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(),
ddl.c_str(), policy.c_str());
+util.execute(sql); util.query(hawq::test::stringFormat("SELECT *
from t_%s", std::to_string(suffix).c_str()), 0);
+
+cmd = hawq::test::stringFormat("hawq register -d %s -f
%s/usage1dat.paq -e 596 t_%s", HAWQ_DB, getHdfsLocation().c_str(),
std::to_string(suffix).c_str());
+EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+util.query(hawq::test::stringFormat("select * from t_%s;",
std::to_string(suffix).c_str()), 100);
+util.execute(hawq::test::stringFormat("insert into t_%s
values(101);", std::to_string(suffix).c_str()));
+util.query(hawq::test::stringFormat("select * from t_%s;",
std::to_string(suffix).c_str()), 101);
+
+suffix ++;
+ }
+}
+
+// hawq register -d hawq_feature_test -f
hdfs://localhost:8020/usage1tmp/ t_#
+for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+auto cmd = hawq::test::stringFormat("hdfs dfs -mkdir -p
%s/usage1tmp/", getHdfsLocation().c_str());
+EXPECT_EQ(0, Command::getCommandStatus(cmd));
+cmd = hawq::test::stringFormat("hdfs dfs -put -f %s/*.paq
%s/usage1tmp/", folderPath.c_str(),
[GitHub] incubator-hawq pull request #880: HAWQ-1037. Modify way to get HDFS port in ...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/880 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #880: HAWQ-1037. Modify way to get HDFS port in ...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/880 HAWQ-1037. Modify way to get HDFS port in TestHawqRegister You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1037 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/880.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #880 commit caaace3cb340dbeb41da1eaed02f0aa5152c0b38 Author: Chunling Wang Date: 2016-08-31T10:19:05Z HAWQ-1037. Modify way to get HDFS port in TestHawqRegister --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #865: HAWQ-1022. Bugfix for getCommandOutput: using ref...
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/865 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #860: HAWQ-1020. Fix bugs to let feature tests T...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/860 HAWQ-1020. Fix bugs to let feature tests TestCommonLib.TestHdfsConfig⦠⦠and TestCommonLib.TestYanConfig run in concourse You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1020 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/860.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #860 commit 1b2e004834ff2f71596ebd892008e6a63a70aaea Author: Chunling Wang Date: 2016-08-25T06:11:16Z HAWQ-1020. Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and TestCommonLib.TestYanConfig run in concourse --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...
Github user wcl14 closed the pull request at: https://github.com/apache/incubator-hawq/pull/828 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/828 HAWQ-969. Add getting configuration from HDFS and YARN Add getting configuration from HDFS and YARN, and writing xml file in xml_parser.cpp. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-969 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/828.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #828 commit f892618a5c3e69f59d8b683352141ed67b478d89 Author: Chunling Wang Date: 2016-08-01T06:59:17Z HAWQ-969. Add getting configuration from HDFS and YARN --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #810: README.md for feature test
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/810 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #801: HAWQ-933. Warning fix for gpnetbench.
Github user wcl14 commented on the issue: https://github.com/apache/incubator-hawq/pull/801 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
