After checking the strace output I notice that the issue was caused by the file mode for files created on vboxsf mount. By default (when automount option is used in virtualbox configuration) it creates .attach_pid<pid> with file mode 0770. When I manually mounted shared folder specifying fmode=660 jstack worked as expected.
Thank you for pointing me to the right path to narrow down the issue. BTW, where I could find the JVM code responsible for handling SIGQUIT signal to see what it does that file mode matters? On Fri, Nov 22, 2013 at 10:30 AM, Piotr Bzdyl <pi...@bzdyl.net> wrote: > Thank you Mikael for the detailed information - I learned a lot about how > SA tools attache to the target process. > > When I was running the test you suggested I noticed one "detail" in my > setup which turned out to be the cause of my problems: as I mentioned > previously I was running my tests inside a virtual machine (namely > VirtualBox). The issue was that the cwd dir of my target process was on the > mounted shared directory (vboxsf filesystem) in VirtualBox VM and it was > causing the problems. I still have to go through the strace outputs to find > out if it is because of the access rights of because how vboxsf file system > driver works (attached the output). > > > On Fri, Nov 22, 2013 at 9:51 AM, Mikael Gerdin > <mikael.ger...@oracle.com>wrote: > >> Piotr, >> >> On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote: >> > Is there any other information I could collect to help troubleshoot this >> > problem? How could I check if VM runs with attach disabled? >> >> The comments in >> >> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java >> describe the handshaking involved in dynamically enabling the attach >> mechanism >> on Linux. >> >> The mechanism is basically: >> * The attacher checks for the existence of the socket file >> /tmp/.java_pid${PID} >> * If that does not exist it creates a file .attach_pid${PID} in the target >> process' working directory and sends a SIGQUIT (kill -QUIT) to that >> process. >> * The SIGQUIT handler notices the .attach_pid${PID} file and creates the >> socket at /tmp/.java_pid${PID} >> * The attacher notices the socket file and unlinks the .attach_pid${PID} >> file. >> >> You can force the target process to always create the socket file at >> startup >> by using the -XX:+StartAttachListener command line flag. >> >> To further troubleshoot the problem I suggest that you try strace:ing the >> target process and jstack with the following flags enabled: >> >> (target process) >> $ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java donothing >> >> (jstack process) >> $ strace -f -e trace=open,connect,stat,unlink,write,kill >> $JAVA_HOME/bin/jstack >> <pid> >> >> You should get something similar to the following: >> (target process) >> (...) >> [pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=12687, >> si_uid=726} --- >> [pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0, >> ...}) = >> 0 >> Process 12733 attached >> [pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5 >> [pid 12733] bind(5, {sa_family=AF_LOCAL, >> sun_path="/tmp/.java_pid12602.tmp"}, >> 110) = 0 >> [pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6 >> >> (jstack process) >> [pid 12688] stat("/localhome/java/jdk-8-ea-bin- >> b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755, st_size=16029, >> ...}) >> = 0 >> [pid 12688] open("/localhome/java/jdk-8-ea-bin- >> b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7 >> [pid 12688] open("/localhome/java/jdk-8-ea-bin- >> b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7 >> [pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT (No >> such >> file or directory) >> [pid 12688] open("/proc/12602/cwd/.attach_pid12602", >> O_RDWR|O_CREAT|O_EXCL, >> 0666) = 7 >> [pid 12688] kill(12602, SIGQUIT) = 0 >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0, >> ...}) = 0 >> [pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0 >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0, >> ...}) = 0 >> [pid 12688] connect(7, {sa_family=AF_LOCAL, >> sun_path="/tmp/.java_pid12602"}, >> 110) = 0 >> [pid 12688] connect(7, {sa_family=AF_LOCAL, >> sun_path="/tmp/.java_pid12602"}, >> 110) = 0 >> [pid 12688] write(7, "1", 1) = 1 >> [pid 12688] write(7, "\0", 1) = 1 >> [pid 12688] write(7, "threaddump", 10) = 10 >> [pid 12688] write(7, "\0", 1) = 1 >> >> /Mikael >> >> > >> > I noticed the same behavior when trying to connect to the process using >> > JConsole: JConsole displays the sample app in the list and when I try to >> > connect it finally displays an error that it cannot connect whereas the >> > sample app prints thread dump on its console. >> > >> > On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman >> <alan.bate...@oracle.com>wrote: >> > > On 21/11/2013 14:47, David Holmes wrote: >> > >> I can recreate this using the minimal VM, which doesn't support >> > >> libattach. So it definitely seems like something is going wrong in >> the >> > >> attach process, but I can't say what. Not sure what debugging hooks >> we >> > >> have >> > >> for the attach mechanism ?? >> > > >> > > The first attach involves signaling the target VM to start up the >> attach >> > > mechanism. So for the minimal VM or a VM running with attach disabled >> (or >> > > as Staffan pointed out, running with a different tmp directory) then >> the >> > > target VM gets a SIGQUIT and just prints the thread dump. It's hard >> to say >> > > what Piotr is running into but I think this at least explains it for >> the >> > > minimal VM. >> > > >> > > -Alan >> >> >
