On Friday 22 November 2013 12.59.42 Piotr Bzdyl wrote: > After checking the strace output I notice that the issue was caused by the > file mode for files created on vboxsf mount. By default (when automount > option is used in virtualbox configuration) it creates .attach_pid<pid> > with file mode 0770. When I manually mounted shared folder specifying > fmode=660 jstack worked as expected. > > Thank you for pointing me to the right path to narrow down the issue. > > BTW, where I could find the JVM code responsible for handling SIGQUIT > signal to see what it does that file mode matters?
http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/os/linux/vm/attachListener_linux.cpp and http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/share/vm/runtime/os.cpp (see signal_thread_entry and its call to AttachListener::is_init_trigger() /Mikael > > On Fri, Nov 22, 2013 at 10:30 AM, Piotr Bzdyl <pi...@bzdyl.net> wrote: > > Thank you Mikael for the detailed information - I learned a lot about how > > SA tools attache to the target process. > > > > When I was running the test you suggested I noticed one "detail" in my > > setup which turned out to be the cause of my problems: as I mentioned > > previously I was running my tests inside a virtual machine (namely > > VirtualBox). The issue was that the cwd dir of my target process was on > > the > > mounted shared directory (vboxsf filesystem) in VirtualBox VM and it was > > causing the problems. I still have to go through the strace outputs to > > find > > out if it is because of the access rights of because how vboxsf file > > system > > driver works (attached the output). > > > > On Fri, Nov 22, 2013 at 9:51 AM, Mikael Gerdin <mikael.ger...@oracle.com>wrote: > >> Piotr, > >> > >> On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote: > >> > Is there any other information I could collect to help troubleshoot > >> > this > >> > problem? How could I check if VM runs with attach disabled? > >> > >> The comments in > >> > >> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun > >> /tools/attach/LinuxVirtualMachine.java describe the handshaking involved > >> in dynamically enabling the attach mechanism > >> on Linux. > >> > >> The mechanism is basically: > >> * The attacher checks for the existence of the socket file > >> /tmp/.java_pid${PID} > >> * If that does not exist it creates a file .attach_pid${PID} in the > >> target > >> process' working directory and sends a SIGQUIT (kill -QUIT) to that > >> process. > >> * The SIGQUIT handler notices the .attach_pid${PID} file and creates the > >> socket at /tmp/.java_pid${PID} > >> * The attacher notices the socket file and unlinks the .attach_pid${PID} > >> file. > >> > >> You can force the target process to always create the socket file at > >> startup > >> by using the -XX:+StartAttachListener command line flag. > >> > >> To further troubleshoot the problem I suggest that you try strace:ing the > >> target process and jstack with the following flags enabled: > >> > >> (target process) > >> $ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java > >> donothing > >> > >> (jstack process) > >> $ strace -f -e trace=open,connect,stat,unlink,write,kill > >> $JAVA_HOME/bin/jstack > >> <pid> > >> > >> You should get something similar to the following: > >> (target process) > >> (...) > >> [pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=12687, > >> si_uid=726} --- > >> [pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0, > >> ...}) = > >> 0 > >> Process 12733 attached > >> [pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5 > >> [pid 12733] bind(5, {sa_family=AF_LOCAL, > >> sun_path="/tmp/.java_pid12602.tmp"}, > >> 110) = 0 > >> [pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6 > >> > >> (jstack process) > >> [pid 12688] stat("/localhome/java/jdk-8-ea-bin- > >> b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755, st_size=16029, > >> ...}) > >> = 0 > >> [pid 12688] open("/localhome/java/jdk-8-ea-bin- > >> b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7 > >> [pid 12688] open("/localhome/java/jdk-8-ea-bin- > >> b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7 > >> [pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT (No > >> such > >> file or directory) > >> [pid 12688] open("/proc/12602/cwd/.attach_pid12602", > >> O_RDWR|O_CREAT|O_EXCL, > >> 0666) = 7 > >> [pid 12688] kill(12602, SIGQUIT) = 0 > >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, > >> st_size=0, > >> ...}) = 0 > >> [pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0 > >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, > >> st_size=0, > >> ...}) = 0 > >> [pid 12688] connect(7, {sa_family=AF_LOCAL, > >> sun_path="/tmp/.java_pid12602"}, > >> 110) = 0 > >> [pid 12688] connect(7, {sa_family=AF_LOCAL, > >> sun_path="/tmp/.java_pid12602"}, > >> 110) = 0 > >> [pid 12688] write(7, "1", 1) = 1 > >> [pid 12688] write(7, "\0", 1) = 1 > >> [pid 12688] write(7, "threaddump", 10) = 10 > >> [pid 12688] write(7, "\0", 1) = 1 > >> > >> /Mikael > >> > >> > I noticed the same behavior when trying to connect to the process using > >> > JConsole: JConsole displays the sample app in the list and when I try > >> > to > >> > connect it finally displays an error that it cannot connect whereas the > >> > sample app prints thread dump on its console. > >> > > >> > On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman > >> > >> <alan.bate...@oracle.com>wrote: > >> > > On 21/11/2013 14:47, David Holmes wrote: > >> > >> I can recreate this using the minimal VM, which doesn't support > >> > >> libattach. So it definitely seems like something is going wrong in > >> > >> the > >> > >> > >> attach process, but I can't say what. Not sure what debugging hooks > >> > >> we > >> > >> > >> have > >> > >> for the attach mechanism ?? > >> > > > >> > > The first attach involves signaling the target VM to start up the > >> > >> attach > >> > >> > > mechanism. So for the minimal VM or a VM running with attach disabled > >> > >> (or > >> > >> > > as Staffan pointed out, running with a different tmp directory) then > >> > >> the > >> > >> > > target VM gets a SIGQUIT and just prints the thread dump. It's hard > >> > >> to say > >> > >> > > what Piotr is running into but I think this at least explains it for > >> > >> the > >> > >> > > minimal VM. > >> > > > >> > > -Alan
