Thank you Mikael. Now with the is_init_trigger() code I understand that it wasn't the problem with file mask but rather with the file owner (lines 504-507 from http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/os/linux/vm/attachListener_linux.cpp ).
What helped was specyfing uid i gid options to mount where provided IDs were IDs of the user running jvm. On Fri, Nov 22, 2013 at 1:17 PM, Mikael Gerdin <mikael.ger...@oracle.com>wrote: > On Friday 22 November 2013 12.59.42 Piotr Bzdyl wrote: > > After checking the strace output I notice that the issue was caused by > the > > file mode for files created on vboxsf mount. By default (when automount > > option is used in virtualbox configuration) it creates .attach_pid<pid> > > with file mode 0770. When I manually mounted shared folder specifying > > fmode=660 jstack worked as expected. > > > > Thank you for pointing me to the right path to narrow down the issue. > > > > BTW, where I could find the JVM code responsible for handling SIGQUIT > > signal to see what it does that file mode matters? > > > http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/os/linux/vm/attachListener_linux.cpp > > and > > > http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/share/vm/runtime/os.cpp > (see signal_thread_entry and its call to AttachListener::is_init_trigger() > > /Mikael > > > > > > On Fri, Nov 22, 2013 at 10:30 AM, Piotr Bzdyl <pi...@bzdyl.net> wrote: > > > Thank you Mikael for the detailed information - I learned a lot about > how > > > SA tools attache to the target process. > > > > > > When I was running the test you suggested I noticed one "detail" in my > > > setup which turned out to be the cause of my problems: as I mentioned > > > previously I was running my tests inside a virtual machine (namely > > > VirtualBox). The issue was that the cwd dir of my target process was on > > > the > > > mounted shared directory (vboxsf filesystem) in VirtualBox VM and it > was > > > causing the problems. I still have to go through the strace outputs to > > > find > > > out if it is because of the access rights of because how vboxsf file > > > system > > > driver works (attached the output). > > > > > > On Fri, Nov 22, 2013 at 9:51 AM, Mikael Gerdin > <mikael.ger...@oracle.com>wrote: > > >> Piotr, > > >> > > >> On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote: > > >> > Is there any other information I could collect to help troubleshoot > > >> > this > > >> > problem? How could I check if VM runs with attach disabled? > > >> > > >> The comments in > > >> > > >> > http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun > > >> /tools/attach/LinuxVirtualMachine.java describe the handshaking > involved > > >> in dynamically enabling the attach mechanism > > >> on Linux. > > >> > > >> The mechanism is basically: > > >> * The attacher checks for the existence of the socket file > > >> /tmp/.java_pid${PID} > > >> * If that does not exist it creates a file .attach_pid${PID} in the > > >> target > > >> process' working directory and sends a SIGQUIT (kill -QUIT) to that > > >> process. > > >> * The SIGQUIT handler notices the .attach_pid${PID} file and creates > the > > >> socket at /tmp/.java_pid${PID} > > >> * The attacher notices the socket file and unlinks the > .attach_pid${PID} > > >> file. > > >> > > >> You can force the target process to always create the socket file at > > >> startup > > >> by using the -XX:+StartAttachListener command line flag. > > >> > > >> To further troubleshoot the problem I suggest that you try strace:ing > the > > >> target process and jstack with the following flags enabled: > > >> > > >> (target process) > > >> $ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java > > >> donothing > > >> > > >> (jstack process) > > >> $ strace -f -e trace=open,connect,stat,unlink,write,kill > > >> $JAVA_HOME/bin/jstack > > >> <pid> > > >> > > >> You should get something similar to the following: > > >> (target process) > > >> (...) > > >> [pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, > si_pid=12687, > > >> si_uid=726} --- > > >> [pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0, > > >> ...}) = > > >> 0 > > >> Process 12733 attached > > >> [pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5 > > >> [pid 12733] bind(5, {sa_family=AF_LOCAL, > > >> sun_path="/tmp/.java_pid12602.tmp"}, > > >> 110) = 0 > > >> [pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6 > > >> > > >> (jstack process) > > >> [pid 12688] stat("/localhome/java/jdk-8-ea-bin- > > >> b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755, > st_size=16029, > > >> ...}) > > >> = 0 > > >> [pid 12688] open("/localhome/java/jdk-8-ea-bin- > > >> b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7 > > >> [pid 12688] open("/localhome/java/jdk-8-ea-bin- > > >> b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7 > > >> [pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT > (No > > >> such > > >> file or directory) > > >> [pid 12688] open("/proc/12602/cwd/.attach_pid12602", > > >> O_RDWR|O_CREAT|O_EXCL, > > >> 0666) = 7 > > >> [pid 12688] kill(12602, SIGQUIT) = 0 > > >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, > > >> st_size=0, > > >> ...}) = 0 > > >> [pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0 > > >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, > > >> st_size=0, > > >> ...}) = 0 > > >> [pid 12688] connect(7, {sa_family=AF_LOCAL, > > >> sun_path="/tmp/.java_pid12602"}, > > >> 110) = 0 > > >> [pid 12688] connect(7, {sa_family=AF_LOCAL, > > >> sun_path="/tmp/.java_pid12602"}, > > >> 110) = 0 > > >> [pid 12688] write(7, "1", 1) = 1 > > >> [pid 12688] write(7, "\0", 1) = 1 > > >> [pid 12688] write(7, "threaddump", 10) = 10 > > >> [pid 12688] write(7, "\0", 1) = 1 > > >> > > >> /Mikael > > >> > > >> > I noticed the same behavior when trying to connect to the process > using > > >> > JConsole: JConsole displays the sample app in the list and when I > try > > >> > to > > >> > connect it finally displays an error that it cannot connect whereas > the > > >> > sample app prints thread dump on its console. > > >> > > > >> > On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman > > >> > > >> <alan.bate...@oracle.com>wrote: > > >> > > On 21/11/2013 14:47, David Holmes wrote: > > >> > >> I can recreate this using the minimal VM, which doesn't support > > >> > >> libattach. So it definitely seems like something is going wrong > in > > >> > > >> the > > >> > > >> > >> attach process, but I can't say what. Not sure what debugging > hooks > > >> > > >> we > > >> > > >> > >> have > > >> > >> for the attach mechanism ?? > > >> > > > > >> > > The first attach involves signaling the target VM to start up the > > >> > > >> attach > > >> > > >> > > mechanism. So for the minimal VM or a VM running with attach > disabled > > >> > > >> (or > > >> > > >> > > as Staffan pointed out, running with a different tmp directory) > then > > >> > > >> the > > >> > > >> > > target VM gets a SIGQUIT and just prints the thread dump. It's > hard > > >> > > >> to say > > >> > > >> > > what Piotr is running into but I think this at least explains it > for > > >> > > >> the > > >> > > >> > > minimal VM. > > >> > > > > >> > > -Alan > >
