[
https://issues.apache.org/jira/browse/FLUME-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292075#comment-13292075
]
Hari Shreedharan commented on FLUME-1248:
-----------------------------------------
Hi Will,
Could you let me know if you have a patch for this yet? I guess it is feasible
for now to use "head -n 1" and note that some gc options can cause problems.
As for the above, I'd rather have the changes go into the Java code than the
already fragile script. This is just a suggestion, please feel free to choose
the one which you think is better.
Thanks.
> flume-ng script gets broken when it tried to load hbase classpath
> -----------------------------------------------------------------
>
> Key: FLUME-1248
> URL: https://issues.apache.org/jira/browse/FLUME-1248
> Project: Flume
> Issue Type: Bug
> Components: Shell
> Affects Versions: v1.1.0
> Reporter: Mingjie Lai
> Assignee: Will McQueen
> Fix For: v1.2.0
>
> Attachments: FLUME-1248.patch
>
>
> bin/flume-ng tried to load hbase/hadoop class path by this:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path 2>/dev/null)
> {code}
> It actually turned out to be:
> {code}
> $ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar \
> org.apache.flume.tools.GetJavaProperty java.library.path
> {code}
> However what I saw is:
> {code}
> -bash-3.2$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar
> org.apache.flume.tools.GetJavaProperty java.library.path
> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64
> Heap
> par new generation total 235968K, used 8391K [0x00000002fae00000,
> 0x000000030ae00000, 0x000000030ae00000)
> eden space 209792K, 4% used [0x00000002fae00000, 0x00000002fb631f30,
> 0x0000000307ae0000)
> from space 26176K, 0% used [0x0000000307ae0000, 0x0000000307ae0000,
> 0x0000000309470000)
> to space 26176K, 0% used [0x0000000309470000, 0x0000000309470000,
> 0x000000030ae00000)
> concurrent mark-sweep generation total 20709376K, used 0K
> [0x000000030ae00000, 0x00000007fae00000, 0x00000007fae00000)
> concurrent-mark-sweep perm gen total 21248K, used 2724K [0x00000007fae00000,
> 0x00000007fc2c0000, 0x0000000800000000)
> {code}
> The hbase gc info outputs to stdout and screwed up the flume-ng script.
> The root cause is the combination of several factors:
> 1. turn on hbase gc log by:
> {code}
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateS
> tamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> {code}
> 2. the gc log directory is protected by limiting the permission as 755, and
> owned by hbase user.
> 3. use another user, such as flume, to execute the script.
> Since flume user doesn't have write permission to the hbase gc log directory,
> jvm will output the gc info to stdout, and the flume script will be screwed
> up.
> A simple but tricky fix could be adding ``grep hbase'' in the scrip to filter
> out the gc info:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path | grep hbase 2>/dev/null)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
