[
https://issues.apache.org/jira/browse/FLUME-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingjie Lai updated FLUME-1248:
-------------------------------
Description:
bin/flume-ng tried to load hbase/hadoop class path by this:
{code}
103 local HBASE_CLASSPATH=""
104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
106 java.library.path 2>/dev/null)
{code}
It actually turned out to be:
{code}
$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar \
org.apache.flume.tools.GetJavaProperty java.library.path
{code}
However what I saw is:
{code}
-bash-3.2$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar
org.apache.flume.tools.GetJavaProperty java.library.path
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64
Heap
par new generation total 235968K, used 8391K [0x00000002fae00000,
0x000000030ae00000, 0x000000030ae00000)
eden space 209792K, 4% used [0x00000002fae00000, 0x00000002fb631f30,
0x0000000307ae0000)
from space 26176K, 0% used [0x0000000307ae0000, 0x0000000307ae0000,
0x0000000309470000)
to space 26176K, 0% used [0x0000000309470000, 0x0000000309470000,
0x000000030ae00000)
concurrent mark-sweep generation total 20709376K, used 0K [0x000000030ae00000,
0x00000007fae00000, 0x00000007fae00000)
concurrent-mark-sweep perm gen total 21248K, used 2724K [0x00000007fae00000,
0x00000007fc2c0000, 0x0000000800000000)
{code}
The hbase gc info outputs to stdout and screwed up the flume-ng script.
The root cause is the combination of several factors:
1. turn on hbase gc log by:
{code}
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateS
tamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
{code}
2. the gc log directory is protected by limiting the permission as 755, and
owned by hbase user.
3. use another user, such as flume, to execute the script.
Since flume user doesn't have write permission to the hbase gc log directory,
jvm will output the gc info to stdout, and the flume script will be screwed up.
A simple but tricky fix could be adding ``grep hbase'' in the scrip to filter
out the gc info:
{code}
103 local HBASE_CLASSPATH=""
104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
106 java.library.path | grep hbase 2>/dev/null)
{code}
was:
bin/flume-ng tried to load hbase/hadoop class path by this:
{code}
103 local HBASE_CLASSPATH=""
104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
106 java.library.path 2>/dev/null)
{code}
It actually turned out to be:
{code}
$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar \
org.apache.flume.tools.GetJavaProperty java.library.path
{code}
However what I saw is:
{code}
-bash-3.2$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar
org.apache.flume.tools.GetJavaProperty java.library.path
/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64
Heap
par new generation total 235968K, used 8391K [0x00000002fae00000,
0x000000030ae00000, 0x000000030ae00000)
eden space 209792K, 4% used [0x00000002fae00000, 0x00000002fb631f30,
0x0000000307ae0000)
from space 26176K, 0% used [0x0000000307ae0000, 0x0000000307ae0000,
0x0000000309470000)
to space 26176K, 0% used [0x0000000309470000, 0x0000000309470000,
0x000000030ae00000)
concurrent mark-sweep generation total 20709376K, used 0K [0x000000030ae00000,
0x00000007fae00000, 0x00000007fae00000)
concurrent-mark-sweep perm gen total 21248K, used 2724K [0x00000007fae00000,
0x00000007fc2c0000, 0x0000000800000000)
{code}
The hbase gc info outputs to stdout and screwed out the flume-ng script.
The root cause is the combination of several factors:
1. turn on hbase gc log by:
{code}
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateS
tamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
{code}
2. the gc log directory is protected by limiting the permission as 755, and
owned by hbase user.
3. use another user, such as flume, to execute the script.
Since flume user doesn't have write permission to the hbase gc log directory,
jvm will output the gc info to stdout, and the flume script will be screwed up.
A simple but tricky fix could be adding ``grep hbase'' in the scrip to filter
out the gc info:
{code}
103 local HBASE_CLASSPATH=""
104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
106 java.library.path | grep hbase 2>/dev/null)
{code}
> flume-ng script gets broken when it tried to load hbase classpath
> -----------------------------------------------------------------
>
> Key: FLUME-1248
> URL: https://issues.apache.org/jira/browse/FLUME-1248
> Project: Flume
> Issue Type: Bug
> Components: Shell
> Affects Versions: v1.1.0
> Reporter: Mingjie Lai
> Fix For: v1.2.0
>
>
> bin/flume-ng tried to load hbase/hadoop class path by this:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path 2>/dev/null)
> {code}
> It actually turned out to be:
> {code}
> $ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar \
> org.apache.flume.tools.GetJavaProperty java.library.path
> {code}
> However what I saw is:
> {code}
> -bash-3.2$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar
> org.apache.flume.tools.GetJavaProperty java.library.path
> /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64
> Heap
> par new generation total 235968K, used 8391K [0x00000002fae00000,
> 0x000000030ae00000, 0x000000030ae00000)
> eden space 209792K, 4% used [0x00000002fae00000, 0x00000002fb631f30,
> 0x0000000307ae0000)
> from space 26176K, 0% used [0x0000000307ae0000, 0x0000000307ae0000,
> 0x0000000309470000)
> to space 26176K, 0% used [0x0000000309470000, 0x0000000309470000,
> 0x000000030ae00000)
> concurrent mark-sweep generation total 20709376K, used 0K
> [0x000000030ae00000, 0x00000007fae00000, 0x00000007fae00000)
> concurrent-mark-sweep perm gen total 21248K, used 2724K [0x00000007fae00000,
> 0x00000007fc2c0000, 0x0000000800000000)
> {code}
> The hbase gc info outputs to stdout and screwed up the flume-ng script.
> The root cause is the combination of several factors:
> 1. turn on hbase gc log by:
> {code}
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateS
> tamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> {code}
> 2. the gc log directory is protected by limiting the permission as 755, and
> owned by hbase user.
> 3. use another user, such as flume, to execute the script.
> Since flume user doesn't have write permission to the hbase gc log directory,
> jvm will output the gc info to stdout, and the flume script will be screwed
> up.
> A simple but tricky fix could be adding ``grep hbase'' in the scrip to filter
> out the gc info:
> {code}
> 103 local HBASE_CLASSPATH=""
> 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
> 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
> 106 java.library.path | grep hbase 2>/dev/null)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
