MisterRaindrop commented on code in PR #22:
URL: https://github.com/apache/cloudberry-pxf/pull/22#discussion_r2646651518
##########
automation/src/main/java/org/greenplum/pxf/automation/utils/system/ProtocolUtils.java:
##########
@@ -23,15 +30,20 @@ public static ProtocolEnum getProtocol() {
}
public static String getSecret() {
- return System.getProperty(AWS_SECRET_ACCESS_KEY);
+ String secret = System.getProperty(AWS_SECRET_ACCESS_KEY);
+ return secret != null ? secret : System.getenv(AWS_SECRET_ACCESS_KEY);
}
public static String getAccess() {
- return System.getProperty(AWS_ACCESS_KEY_ID);
+ String access = System.getProperty(AWS_ACCESS_KEY_ID);
+ String result = access != null ? access :
System.getenv(AWS_ACCESS_KEY_ID);
+ System.out.println("DEBUG: doInit() method called getAccess " +
result);
Review Comment:
this is a debug print?
##########
concourse/singlecluster/conf/gphd-conf.sh:
##########
@@ -4,14 +4,7 @@ if [ -z "${JAVA_HOME:-}" ]; then
if [ -n "${JAVA_HADOOP:-}" ]; then
export JAVA_HOME="${JAVA_HADOOP}"
else
- # Auto-detect Java 8 path for different architectures
- if [ -d "/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" ]; then
- export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-$(dpkg
--print-architecture)"
- elif [ -d "/usr/lib/jvm/java-8-openjdk" ]; then
- export JAVA_HOME="/usr/lib/jvm/java-8-openjdk"
- else
- export JAVA_HOME=$(readlink -f /usr/bin/java | sed 's:/bin/java::')
- fi
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk
Review Comment:
delete Auto-detect Java 8 path for different architectures？
##########
automation/src/test/java/org/greenplum/pxf/automation/features/hive/HiveBaseTest.java:
##########
@@ -371,25 +371,11 @@ void loadDataIntoHive(String fileName, HiveTable
tableName) throws Exception {
void loadDataIntoHive(Hdfs hdfs, Hive hive, String fileName, HiveTable
tableName) throws Exception {
- String localPath = localDataResourcesFolder + "/hive/" + fileName;
- String hdfsPath = hdfs.getWorkingDirectory() + "/" + fileName;
-
- // Verify local file exists
- java.io.File localFile = new java.io.File(localPath);
- if (!localFile.exists()) {
- throw new RuntimeException("Local file does not exist: " +
localFile.getAbsolutePath());
- }
-
// copy data to hdfs
- hdfs.copyFromLocal(localPath, hdfsPath);
-
- // Verify file was copied to HDFS
- if (!hdfs.doesFileExist(hdfsPath)) {
- throw new RuntimeException("File was not copied to HDFS: " +
hdfsPath);
- }
-
+ hdfs.copyFromLocal(localDataResourcesFolder + "/hive/" + fileName,
+ hdfs.getWorkingDirectory() + "/" + fileName);
// load to hive table
- hive.loadData(tableName, hdfsPath, false);
+ hive.loadData(tableName, hdfs.getWorkingDirectory() + "/" + fileName,
false);
Review Comment:
Sometimes HDFS metadata loading isn't that fast. Would it be better to add
hdfs.waitForFile? ensure data load successfully?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
