[GitHub] incubator-hawq pull request #1390: HAWQ-1650. Fix compilation issue in Java ...
GitHub user frankgh opened a pull request: https://github.com/apache/incubator-hawq/pull/1390 HAWQ-1650. Fix compilation issue in Java 7 Fixes compilation issue in Java 7 for SecureHDFSTest You can merge this pull request into a Git repository by running: $ git pull https://github.com/frankgh/incubator-hawq HAWQ-1650 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1390.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1390 commit 9f33d8dd0c97d06a5e2fcb555b73fd83281c1494 Author: Francisco Guerrero Date: 2018-08-13T21:43:06Z HAWQ-1650. Fix compilation issue in Java 7 ---
[GitHub] incubator-hawq pull request #1389: Run gradle build in parallel
GitHub user frankgh opened a pull request: https://github.com/apache/incubator-hawq/pull/1389 Run gradle build in parallel Improve PXF compilation time by running gradle in parallel and using gradle daemon You can merge this pull request into a Git repository by running: $ git pull https://github.com/frankgh/incubator-hawq HAWQ-1649 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1389.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1389 commit 2c4af56f407fb65eb7bdc7b690984907c0d2c36b Author: Francisco Guerrero Date: 2018-08-13T18:45:14Z Add parallel true and daemon to gradle properties ---
[GitHub] incubator-hawq issue #1386: HAWQ-1646. Fixes travis CI issues
Github user frankgh commented on the issue: https://github.com/apache/incubator-hawq/pull/1386 @shivzone @lavjain @divyabhargov This PR addresses travis-ci issues. ---
[GitHub] incubator-hawq pull request #1386: HAWQ-1646. Fixes travis CI issues
GitHub user frankgh opened a pull request: https://github.com/apache/incubator-hawq/pull/1386 HAWQ-1646. Fixes travis CI issues Travis CI is using a new default OSX image starting July 31st, which is causing compilation issues in javadocs. This PR sets the previously known working osx_image in .travis.yml. You can merge this pull request into a Git repository by running: $ git pull https://github.com/frankgh/incubator-hawq HAWQ-1646 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1386.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1386 commit 5b185559814a86b66ba29544a5ebe2d71dc8 Author: Francisco Guerrero Date: 2018-08-07T06:14:27Z HAWQ-1646. Fixes travis CI issues ---
[GitHub] incubator-hawq pull request #1385: HAWQ-1645. Remove code generation from PX...
GitHub user frankgh opened a pull request: https://github.com/apache/incubator-hawq/pull/1385 HAWQ-1645. Remove code generation from PXF and upgrade gradle version - Removes code generation from PXF to enable IntelliJ code intelligence - Upgrade gradle to version 4.9 You can merge this pull request into a Git repository by running: $ git pull https://github.com/frankgh/incubator-hawq HAWQ-1645 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1385.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1385 commit 41eec36532c3e5a2bb6fb5174b099f23669e87a2 Author: Pivotal Date: 2018-08-06T22:15:05Z HAWQ-1645. Use the latest available version of gradle (4.9) ---
[GitHub] incubator-hawq pull request #1379: HAWQ-1622. Cache PXF proxy UGI so that cl...
Github user frankgh commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r202409510
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/UGICache.java ---
@@ -0,0 +1,318 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.DelayQueue;
+import java.util.concurrent.Delayed;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.base.Ticker;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.security.UserGroupInformation;
+
+/**
+ * Stores UserGroupInformation instances for each active session. The UGIs
are cleaned up if they
+ * have not been accessed for 15 minutes.
+ *
+ * The motivation for caching is that destroying UGIs is slow. The
alternative, creating and
+ * destroying a UGI per-request, is wasteful.
+ */
+public class UGICache {
+
+private static final Log LOG = LogFactory.getLog(UGICache.class);
+private Map cache = new ConcurrentHashMap<>();
+@SuppressWarnings("unchecked")
+// There is a separate DelayQueue for each segment (also being used
for locking)
+private final Map> queueMap = new
HashMap<>();
+private final UGIProvider ugiProvider;
+private Ticker ticker;
+private final static long UGI_CACHE_EXPIRY = 15 * 60 * 1000L; // 15
Minutes
+
+/**
+ * Create a UGICache with the given {@link UGIProvider}. Intended for
use by tests which need
+ * to substitute a mock UGIProvider.
+ */
+UGICache(UGIProvider provider, Ticker ticker) {
+this.ticker = ticker;
+this.ugiProvider = provider;
+}
+
+/**
+ * Create a UGICache. Automatically creates a {@link UGIProvider} that
this cache will use to
+ * create and destroy UserGroupInformation instances.
+ */
+public UGICache() {
+this(new UGIProvider(), Ticker.systemTicker());
+}
+
+/**
+ * Create new proxy UGI if not found in cache and increment reference
count
+ */
+public UserGroupInformation getUserGroupInformation(SessionId session)
throws IOException {
+Integer segmentId = session.getSegmentId();
+String user = session.getUser();
+DelayQueue delayQueue = getExpirationQueue(segmentId);
+synchronized (delayQueue) {
+// Use the opportunity to cleanup any expired entries
+cleanup(session);
+Entry entry = cache.get(session);
+if (entry == null) {
+LOG.info(session.toString() + " Creating proxy user = " +
user);
+entry = new Entry(ticker,
ugiProvider.createProxyUGI(user));
+delayQueue.offer(entry);
+cache.put(session, entry);
+}
+entry.acquireReference();
+return entry.getUGI();
+}
+}
+
+/**
+ * Decrement reference count for the given session's UGI. Resets the
time at which the UGI will
+ * expire to 15 minutes in the future.
+ *
+ * @param session the session for which we want to
release the UGI.
+ * @param cleanImmediatelyIfNoRefs if true, destroys the UGI for the
given session (only if it is
+ * now unreferenced).
+ */
+public void release(SessionId session, boolean
cleanImmediatelyIfNoRefs) {
+
+Entry timedProxyUGI = cache.get(session);
+
+if (timedProxyUGI == null) return;
+
+timedProxy
[GitHub] incubator-hawq pull request #1379: HAWQ-1622. Cache PXF proxy UGI so that cl...
Github user frankgh commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r200791461
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/SegmentTransactionId.java
---
@@ -0,0 +1,60 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+public class SegmentTransactionId {
+
+private Integer segmentId;
+private String transactionId;
--- End diff --
`transactionId` is not being used anywhere, it can be removed.
---
[GitHub] incubator-hawq pull request #1379: HAWQ-1622. Cache PXF proxy UGI so that cl...
Github user frankgh commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r200793645
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/UGICache.java ---
@@ -0,0 +1,129 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.DelayQueue;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.security.UserGroupInformation;
+
+public class UGICache {
+
+private static final Log LOG = LogFactory.getLog(UGICache.class);
+private static Map cache = new
ConcurrentHashMap<>();
+//private static DelayQueue delayQueue = new
DelayQueue<>();
+private static DelayQueue[] delayQueues = new
DelayQueue<>[64];
+public static long UGI_CACHE_EXPIRY = 15 * 1 * 1000L; // 15 Minutes
+
+public UGICache() {
+for (int i = 0; i < delayQueues.length; i++) {
+delayQueues[i] = new DelayQueue<>();
--- End diff --
delayQueues is static. Every time a UGICache is instantiated, the
`delayQueues` array will be initialized. We may want to initialize in a static
ctor.
---
[GitHub] incubator-hawq pull request #1379: HAWQ-1622. Cache PXF proxy UGI so that cl...
Github user frankgh commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r200793774
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/UGICache.java ---
@@ -0,0 +1,129 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.DelayQueue;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.security.UserGroupInformation;
+
+public class UGICache {
+
+private static final Log LOG = LogFactory.getLog(UGICache.class);
+private static Map cache = new
ConcurrentHashMap<>();
+//private static DelayQueue delayQueue = new
DelayQueue<>();
+private static DelayQueue[] delayQueues = new
DelayQueue<>[64];
--- End diff --
Are we limiting to 64 segments?
---
[GitHub] incubator-hawq pull request #1379: HAWQ-1622. Cache PXF proxy UGI so that cl...
Github user frankgh commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r200793183
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/UGICache.java ---
@@ -0,0 +1,129 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.DelayQueue;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.security.UserGroupInformation;
+
+public class UGICache {
+
+private static final Log LOG = LogFactory.getLog(UGICache.class);
+private static Map cache = new
ConcurrentHashMap<>();
+//private static DelayQueue delayQueue = new
DelayQueue<>();
+private static DelayQueue[] delayQueues = new
DelayQueue<>[64];
+public static long UGI_CACHE_EXPIRY = 15 * 1 * 1000L; // 15 Minutes
--- End diff --
Can `UGI_CACHE_EXPIRY` be parameterized?
---
[GitHub] incubator-hawq pull request #1379: WIP: Cache UGI objects and clean them per...
Github user frankgh commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1379#discussion_r199547098
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/servlet/SecurityServletFilter.java
---
@@ -42,8 +52,51 @@
private static final Log LOG =
LogFactory.getLog(SecurityServletFilter.class);
private static final String USER_HEADER = "X-GP-USER";
-private static final String MISSING_HEADER_ERROR =
String.format("Header %s is missing in the request", USER_HEADER);
-private static final String EMPTY_HEADER_ERROR = String.format("Header
%s is empty in the request", USER_HEADER);
+private static final String SEGMENT_INDEX_HEADER = "X-GP-SEGMENT-ID";
+private static final String TRANSACTION_ID_HEADER = "X-GP-XID";
+private static final String MISSING_HEADER_ERROR = "Header %s is
missing in the request";
+private static final String EMPTY_HEADER_ERROR = "Header %s is empty
in the request";
+private static Map cache = new
ConcurrentHashMap<>();
--- End diff --
I agree with @shivzone about using guava, especially considering that we
are already using it in this project.
https://github.com/google/guava/wiki/CachesExplained
---
