[
https://issues.apache.org/jira/browse/FLUME-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290905#comment-13290905
]
Will McQueen commented on FLUME-1248:
-------------------------------------
Hi Mingjie,
In the diff you posted from the previously att'd patch, the HADOOP_OPTS and
HBASE_OPTS are set to empty string so that GC options wouldn't be passed to
hbase when invoked from Flume and then the output from the GetJavaProperty call
wouldn't come back with extra lines. Those 2 lines probably should've specified
'export':
export HADOOP_OPTS=""
export HBASE_OPTS=""
...but regardless, I can still reproduce the issue when I uncomment the line in
hbase-env.sh that adds the GC options:
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps"
...so I cancelled the patch and it looks like this needs more discussion (and I
need to test before submitting the next patch).
If we look at hbase script, it's sourcing hbase-config.sh, which in turn
sources hadoop-env.sh, in which a user can specify an HBASE_OPTS value to
startup hbase with. If that HBASE_OPTS value has gc args like "-verbose:gc",
then we know that flume will get back unexpected output. So, I think we should
reconsider your suggestion and possibly modify it. So your suggestion was:
head -n1
..and I believe that it will solve exactly the problem you posted in the
ticket, using the jvm args that you mention. So if we want to just solve the
current problem, then this seems like a straightforward solution. But if we
want a solution that covers additional cases, such as "-verbose:class" option
(or any other option that writes to stdout before the program's main() output
has been written), then I feel we have some choices, including:
1) Call GetJavaProperties w/o any args, which will cause all env vars to be
printed, one per line. So then grep for the line that starts with key
"java.library.path=", and grab the associated value and we're done.
..or..
2) Modify GetJavaProperties so that calling it with an arg would print the key
to stdout, then proceed as in step 1. So change this:
{code}
System.out.println(System.getProperty(prop, ""));
{code}
...to this:
{code}
System.out.println(prop + "=" + System.getProperty(prop, ""));
{code}
#2 will produce a lot less output to parse on each startup of the agent, while
#1 just requires a change to the script and not to the Java code.
Thoughts?
> flume-ng script gets broken when it tried to load hbase classpath
> -----------------------------------------------------------------
>
> Key: FLUME-1248
> URL: https://issues.apache.org/jira/browse/FLUME-1248
> Project: Flume
> Issue Type: Bug
> Components: Shell
> Affects Versions: v1.1.0
> Reporter: Mingjie Lai
> Assignee: Will McQueen
> Fix For: v1.2.0
>
> Attachments: FLUME-1248.patch
>
>
> bin/flume-ng tried to load hbase/hadoop class path by this:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path 2>/dev/null)
> {code}
> It actually turned out to be:
> {code}
> $ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar \
> org.apache.flume.tools.GetJavaProperty java.library.path
> {code}
> However what I saw is:
> {code}
> -bash-3.2$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar
> org.apache.flume.tools.GetJavaProperty java.library.path
> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64
> Heap
> par new generation total 235968K, used 8391K [0x00000002fae00000,
> 0x000000030ae00000, 0x000000030ae00000)
> eden space 209792K, 4% used [0x00000002fae00000, 0x00000002fb631f30,
> 0x0000000307ae0000)
> from space 26176K, 0% used [0x0000000307ae0000, 0x0000000307ae0000,
> 0x0000000309470000)
> to space 26176K, 0% used [0x0000000309470000, 0x0000000309470000,
> 0x000000030ae00000)
> concurrent mark-sweep generation total 20709376K, used 0K
> [0x000000030ae00000, 0x00000007fae00000, 0x00000007fae00000)
> concurrent-mark-sweep perm gen total 21248K, used 2724K [0x00000007fae00000,
> 0x00000007fc2c0000, 0x0000000800000000)
> {code}
> The hbase gc info outputs to stdout and screwed up the flume-ng script.
> The root cause is the combination of several factors:
> 1. turn on hbase gc log by:
> {code}
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateS
> tamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> {code}
> 2. the gc log directory is protected by limiting the permission as 755, and
> owned by hbase user.
> 3. use another user, such as flume, to execute the script.
> Since flume user doesn't have write permission to the hbase gc log directory,
> jvm will output the gc info to stdout, and the flume script will be screwed
> up.
> A simple but tricky fix could be adding ``grep hbase'' in the scrip to filter
> out the gc info:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path | grep hbase 2>/dev/null)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
