[jira] [Commented] (ACCUMULO-4561) Crash when using ping on a non-existing server
[
https://issues.apache.org/jira/browse/ACCUMULO-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201366#comment-16201366
]
Christopher Tubbs commented on ACCUMULO-4561:
-
That's interesting. The thrift client should have a more sane failure behavior
when the TCP response is not in the expected protocol. Jetty is detecting the
protocol mismatch, and reacting reasonably using the only protocol it knows;
there's no reason Thrift shouldn't behave similarly.
For a client, the response should be to throw an Exception if the protocol
doesn't match. For a server, it should respond with an error message in its
native protocol, like Jetty did with the HTTP 400 error.
This might be something we need to escalate to Thrift upstream developers, if
it's not something already built in to the Thrift client that we're not
handling properly. The next step should be to check to see if our client code
is failing to properly handle a relevant exception coming from the Thrift
library.
> Crash when using ping on a non-existing server
> --
>
> Key: ACCUMULO-4561
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4561
> Project: Accumulo
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Luis Tavarez
>
> While working on ACCUMULO-4558, I tried running
> {code}ping -ts localhost:9995{code} (localhost:9995 does not have a a tserver
> on my setup.)
> And it caused the shell to exit (crashed) and show the following message.
> {code}#
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="kill -9 %p"
> # Executing /bin/sh -c "kill -9 25561"...
> /home/lmtavar/git/uno/bin/uno: line 44: 25561 Killed
> "$ACCUMULO_HOME"/bin/accumulo shell -u "$ACCUMULO_USER" -p
> "$ACCUMULO_PASSWORD" "${@:2}"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4561) Crash when using ping on a non-existing server
[
https://issues.apache.org/jira/browse/ACCUMULO-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200641#comment-16200641
]
Mark Owens commented on ACCUMULO-4561:
--
A little more info: The monitor.log returns the following warning when the
ping command targets a Jetty port:
{noformat}
2017-10-11 13:09:25,039 [http.HttpParser] WARN : Illegal character 0x0 in
state=START for buffer
HeapByteBuffer@3249e354[p=1,l=179,c=8192,r=178]={\x00<<<\x00\x00\xAf\x82!\x01\x15getTabletS...d92138d\x00,\x16\x00\x16\x00\x00\x00>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2017-10-11 13:09:25,039 [http.HttpParser] WARN : bad HTTP parsed: 400 Illegal
character 0x0 for HttpChannelOverHttp@4676385e{r=0,c=false,a=IDLE,uri=null}
{noformat}
The returned HTTP payload (via wireshark) is:
{noformat}
48 54 54 50 2f 31 2e 31 20 34 30 30 20 49 6c 6c HTTP/1.1 400 Ill
0010 65 67 61 6c 20 63 68 61 72 61 63 74 65 72 20 30 egal character 0
0020 78 30 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 x0..Content-Leng
0030 74 68 3a 20 30 0d 0a 43 6f 6e 6e 65 63 74 69 6f th: 0..Connectio
0040 6e 3a 20 63 6c 6f 73 65 0d 0a 53 65 72 76 65 72 n: close..Server
0050 3a 20 4a 65 74 74 79 28 39 2e 33 2e 32 31 2e 76 : Jetty(9.3.21.v
0060 32 30 31 37 30 39 31 38 29 0d 0a 0d 0a 20170918)
{noformat}
> Crash when using ping on a non-existing server
> --
>
> Key: ACCUMULO-4561
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4561
> Project: Accumulo
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Luis Tavarez
>
> While working on ACCUMULO-4558, I tried running
> {code}ping -ts localhost:9995{code} (localhost:9995 does not have a a tserver
> on my setup.)
> And it caused the shell to exit (crashed) and show the following message.
> {code}#
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="kill -9 %p"
> # Executing /bin/sh -c "kill -9 25561"...
> /home/lmtavar/git/uno/bin/uno: line 44: 25561 Killed
> "$ACCUMULO_HOME"/bin/accumulo shell -u "$ACCUMULO_USER" -p
> "$ACCUMULO_PASSWORD" "${@:2}"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4561) Crash when using ping on a non-existing server
[
https://issues.apache.org/jira/browse/ACCUMULO-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200561#comment-16200561
]
Keith Turner commented on ACCUMULO-4561:
The ping command makes a thrift connection to the destination. In this case
the destination does not speak thrift and send something back, not sure what.
I suspect the response from Jetty is read by the thrift code and used to try
and allocate a byte array which exceeds available memory.
My memory is fuzzy on this, but I think on the server side in Accumulo we use
the Thrift framed transport and limit the maximum byte array it will allocate.
I think this is configurable. I am not sure if this can be done on the thrift
client side though.
> Crash when using ping on a non-existing server
> --
>
> Key: ACCUMULO-4561
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4561
> Project: Accumulo
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Luis Tavarez
>
> While working on ACCUMULO-4558, I tried running
> {code}ping -ts localhost:9995{code} (localhost:9995 does not have a a tserver
> on my setup.)
> And it caused the shell to exit (crashed) and show the following message.
> {code}#
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="kill -9 %p"
> # Executing /bin/sh -c "kill -9 25561"...
> /home/lmtavar/git/uno/bin/uno: line 44: 25561 Killed
> "$ACCUMULO_HOME"/bin/accumulo shell -u "$ACCUMULO_USER" -p
> "$ACCUMULO_PASSWORD" "${@:2}"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ACCUMULO-4561) Crash when using ping on a non-existing server
[
https://issues.apache.org/jira/browse/ACCUMULO-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200425#comment-16200425
]
Mark Owens commented on ACCUMULO-4561:
--
These crashes appear to be occurring when sending a ping request to ports that
have Jetty listening. I ran an nmap scan on my local machine looking for open
ports and then ran the accumulo shell ping command against the open ports
(closed ports return connection refused).
Note that all these tests were run on the 2.0.0-SNAPSHOT.
My results are listed below:
{{
TServer port on local instance:
9997/tcp open palace-6?
>>> localhost:9997:OK
Following ports all returned same response:
2181/tcp open eforward?
4560/tcp open unknown
5355/tcp open llmnr?
8030/tcp open hadoop-ipc Hadoop IPC
8031/tcp open hadoop-ipc Hadoop IPC
8032/tcp open hadoop-ipc Hadoop IPC
8033/tcp open hadoop-ipc Hadoop IPC
8040/tcp open hadoop-ipc Hadoop IPC
9000/tcp open hadoop-ipc Hadoop IPC
34737/tcp open unknown
39473/tcp open hadoop-ipc Hadoop IPC
50010/tcp open unknown
50020/tcp open hadoop-ipc Hadoop IPC
>>> localhost:8031 ERROR org.apache.thrift.transport.TTransportException
9998/tcp open distinct32?
/tcp open abyss?
10001/tcp open scp-config?
>>> localhost:9998 ERROR org.apache.thrift.TApplicationException: Invalid
>>> method name: 'getTabletServerStatus'
13562/tcp open unknown
>>> localhost:13562 ERROR org.apache.thrift.transport.TTransportException:
>>> java.net.SocketTimeoutException: 12 millis timeout while waiting for
>>> channel to be ready for read. ch :
>>> java.nio.channels.SocketChannel[connected local=/127.0.0.1:36716
>>> remote=localhost/127.0.0.1:13562]
Jetty ports:
8042/tcp open httpJetty 6.1.26
8088/tcp open httpJetty 6.1.26
9995/tcp open httpJetty 9.3.21.v20170918
44263/tcp open httpJetty 6.1.26
50070/tcp open httpJetty 6.1.26
50090/tcp open httpJetty 6.1.26
>>> #
>>> # java.lang.OutOfMemoryError: Java heap space
>>> # -XX:OnOutOfMemoryError="kill -9 %p"
>>> # Executing /bin/sh -c "kill -9 7693"...
>>> Killed
This port returned a different response after a timeout:
50075/tcp open httpJetty 6.1.26
>>> localhost:50075 ERROR org.apache.thrift.transport.TTransportException:
>>> java.net.SocketTimeoutException: 12 millis timeout while waiting for
>>> channel to be ready for read. ch :
>>> java.nio.channels.SocketChannel[connected local=/127.0.0.1:37190
>>> remote=localhost/127.0.0.1:50075]
}}
I have no feel for how often the 'ping -ts ' command is run and how often
it would be provided an invalid port? I would assume a user would only supply a
port if they suspected it to be a tserver. Given that case this situation would
not happen very often, I suspect.
I also noticed that if I stop the tablet servers after I'm in the shell and
then run the ping command, the shell never returns the prompt to the user.
Ctrl-C'ing at that points exits the shell as well. I would think that should
be fixed since the purpose of the ping is to retrieve the status of a tablet
server. Has that behavior been documented and/or verified previously?
> Crash when using ping on a non-existing server
> --
>
> Key: ACCUMULO-4561
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4561
> Project: Accumulo
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Luis Tavarez
>
> While working on ACCUMULO-4558, I tried running
> {code}ping -ts localhost:9995{code} (localhost:9995 does not have a a tserver
> on my setup.)
> And it caused the shell to exit (crashed) and show the following message.
> {code}#
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="kill -9 %p"
> # Executing /bin/sh -c "kill -9 25561"...
> /home/lmtavar/git/uno/bin/uno: line 44: 25561 Killed
> "$ACCUMULO_HOME"/bin/accumulo shell -u "$ACCUMULO_USER" -p
> "$ACCUMULO_PASSWORD" "${@:2}"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
