[
https://issues.apache.org/jira/browse/FLUME-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287298#comment-13287298
]
Will McQueen commented on FLUME-1248:
-------------------------------------
Mingjie,
For 'head -n1', would it be safe to assume that the first line of output would
always contain the desired output of GetJavaProperty? Also, what kind of check
did you have in mind? If the check fails, then would we parse the next line, or
fail?
Mike,
Sounds like you have a good, simple solution for option #2 that doesn't involve
any parsing of output. And I don't think that any files in /etc/default would
get sourced, because I believe those are sourced only from init scripts... so I
think now there should be no concern about getting unexpected output to stdout
since the flume-ng script is calling the hadoop/hbase executable directly. +1
for your solution, unless someone feels a need for wanting to run with JVM
options for GetJavaProperty.
Btw, here's some code for solution #3. It would require a new lib (Apache
commons codec) for DigestUtils, and some flume-ng script changes for doing the
parsing. It would also mean that since the flume-ng script would need use
md5sum, the user must have md5sum installed on their machine. Using your
solution obviates the need for all this.
{code}
public class GetJavaProperty2 {
private static final long DELIM = 0xFEEDDADEEL; //68432014830
public static void main(String args[]) {
args = new String[] {"java.class.path"};
StringBuilder payloadBuilder = new StringBuilder();
if (args.length == 0) {
for (Object prop : System.getProperties().keySet()) {
payloadBuilder.append(prop + "="
+ System.getProperty(prop.toString(), ""));
}
} else {
for (String prop : args) {
payloadBuilder.append(System.getProperty(prop, ""));
}
}
String payload = payloadBuilder.toString();
StringBuilder envelope = new StringBuilder();
envelope.append("||").append(DELIM);
envelope.append("||").append(payload.toString());
envelope.append("||").append(DELIM);
String md5Trailer = DigestUtils.md5Hex(payload.toString());
StringBuilder output = new StringBuilder(envelope);
output.append("||").append(md5Trailer);
output.append("||");
System.out.println(output);
}
}
{code}
> flume-ng script gets broken when it tried to load hbase classpath
> -----------------------------------------------------------------
>
> Key: FLUME-1248
> URL: https://issues.apache.org/jira/browse/FLUME-1248
> Project: Flume
> Issue Type: Bug
> Components: Shell
> Affects Versions: v1.1.0
> Reporter: Mingjie Lai
> Assignee: Will McQueen
> Fix For: v1.2.0
>
>
> bin/flume-ng tried to load hbase/hadoop class path by this:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path 2>/dev/null)
> {code}
> It actually turned out to be:
> {code}
> $ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar \
> org.apache.flume.tools.GetJavaProperty java.library.path
> {code}
> However what I saw is:
> {code}
> -bash-3.2$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar
> org.apache.flume.tools.GetJavaProperty java.library.path
> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64
> Heap
> par new generation total 235968K, used 8391K [0x00000002fae00000,
> 0x000000030ae00000, 0x000000030ae00000)
> eden space 209792K, 4% used [0x00000002fae00000, 0x00000002fb631f30,
> 0x0000000307ae0000)
> from space 26176K, 0% used [0x0000000307ae0000, 0x0000000307ae0000,
> 0x0000000309470000)
> to space 26176K, 0% used [0x0000000309470000, 0x0000000309470000,
> 0x000000030ae00000)
> concurrent mark-sweep generation total 20709376K, used 0K
> [0x000000030ae00000, 0x00000007fae00000, 0x00000007fae00000)
> concurrent-mark-sweep perm gen total 21248K, used 2724K [0x00000007fae00000,
> 0x00000007fc2c0000, 0x0000000800000000)
> {code}
> The hbase gc info outputs to stdout and screwed up the flume-ng script.
> The root cause is the combination of several factors:
> 1. turn on hbase gc log by:
> {code}
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateS
> tamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> {code}
> 2. the gc log directory is protected by limiting the permission as 755, and
> owned by hbase user.
> 3. use another user, such as flume, to execute the script.
> Since flume user doesn't have write permission to the hbase gc log directory,
> jvm will output the gc info to stdout, and the flume script will be screwed
> up.
> A simple but tricky fix could be adding ``grep hbase'' in the scrip to filter
> out the gc info:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path | grep hbase 2>/dev/null)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
