Java Threads on Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Re-Reading the O'Reilly book "Java Threads 2nd Edition by Scott Oaks and Henry Wong", in summary: In Windows: * Threads are timesliced * With SMP, each CPU will select a currently running thread ( page 147 ) ( Therefore, no need to do some native call just to ensure all CPUs are used... even though there is code in the book to call setConcurrency() ). In Solaris: * Two-level thread model ( LWP or system threads and user threads ) * Threads are scheduled based on Solaris LWP. Each LWP ( lightweight process / system level threads ) are known and scheduled by the OS. Each LWP will run the highest priority ( user level ) thread available. * LWPs do not perform timeslicing among the eligible threads ( just like green-threads ) * Because the number of LWPs are controlled by the Solaris thread library, you may not get the "right" number of LWPs if you have, for example, a 8 CPUs ...( page 152 ). The only way to set the number of LWPs is via a native call. I have seen and heard people complaining about why their Java program in Solaris is only using 1 or 2 CPUs, and not using all the available CPUs. I always point them out to that page on the book. In Linux: Well, I hope to hear it from you guys. ;) ... since it is not in the book that I have and I am not a kernel guru ... and how is it different between 2.2 and 2.4. Thanks, John -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.4 (GNU/Linux) Comment: For info see http://www.gnupg.org iEYEARECAAYFAju6t0cACgkQAvd5SY4qWYzFMACeKCdnH21wLxRiUOgiJ/b88ntV PL0AnRWGixy6f8S9we4gotsM9RfDaPwo =Dz0k -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Linux - Java Open Files
Hi I need some clarification on a problem I'm having running java under Linux. I start a number of java applications (JDK 1.3.1_01) on our Red hat 6.2 box. Now as I understand what happens is, when a thread is spawned, each thread is cloned and becomes its own process. This is since JDK 1.3.x, while JDK 1.2.x used green threads. What I need to understand is, do they share the same memory? And the same files? The problem I am having is the number of files that are opened by my applications. Listed using lsof. It seems that each thread (each process) has a handle to each jar file that it references. For example, after I have started all my applications I have about 600 java processes running on the Linux box. Doing the following lsof | grep log4j.jar --count" will return about 580 instances. Doing a complete count on all the files open by the user who starts the application, I get about 55000 open files Any help would be great Thanks in advance Jeff Singer -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
RE: Linux - Java Open Files
Hi Thanks for the reply. Im still not sure exactly what you meant with the open file issue. If the open file is counted one per thread, is the file open for each thread or only reported as open. If so lsof might not be accurate with threads ?? The reason we started looking at the open files is, we where having IO problems. After running for about 1 day, Linix started reporting IO errors and stating that it couldnt open any more files. The other concern or question is, should the handle to a referenced jar file not be closed at some point? Thanks Jeff > -Original Message- > From: Anders Lindback [mailto:[EMAIL PROTECTED]] > Sent: 03 October 2001 12:21 > To: Jeff Singer > Cc: [EMAIL PROTECTED] > Subject: Re: Linux - Java Open Files > > > Jeff Singer skrev: > > I start a number of java applications (JDK 1.3.1_01) on our Red > hat 6.2 box. > > Now as I understand what happens is, when a thread is spawned, > each thread > > is cloned and becomes its own process. This is since JDK 1.3.x, > while JDK > > 1.2.x used green threads. > > Actualyy no they do not become their own process. In the Linux a thread > gets a process entry so that it looks and behaves like a process for > scheduling etc. But a thread is not a process. > > > What I need to understand is, do they share the same memory? > And the same > > files? > > All threads share their memory open files etc thisi whats > separate them from > normal processes. > > > The problem I am having is the number of files that are opened by my > > applications. Listed using lsof. It seems that each thread > (each process) > > has a handle to each jar file that it references. For example, > after I have > > started all my applications I have about 600 java processes > running on the > > Linux box. Doing the following lsof | grep log4j.jar --count" > will return > > about 580 instances. > > > > Doing a complete count on all the files open by the user who starts the > > application, I get about 55000 open files > > Each open files are counted 580 times - one for each thread - thus > the program containging all threads has about 100 open files. > > Anders -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Java Threads on Linux
> In Linux: > Well, I hope to hear it from you guys. ;) ... since it is not in the book > that I have and I am not a kernel guru ... and how is it different between > 2.2 and 2.4. Hello all, I'm not a kernel guru too, but what know is that : java threads are mapped to posix threads (linuxthreads in the glibc). The linuxthreads library uses the kernel threads, which directly map into a 'process like' entry in the scheduler. (1-1 mapping). One has already said that the linux kernel is optimized for the common case, ie few processus in the scheduler queue. m-n mapping may be coming soon on linux, thanks to http://oss.software.ibm.com/developerworks/projects/pthreads I don't know differences between 2.2 and 2.4 Florent -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
RE: Linux - Java Open Files
On Wed, 2001-10-03 at 05:37, Jeff Singer wrote:
>
> Im still not sure exactly what you meant with the open file issue.
> If the open file is counted one per thread, is the file open for each
> thread or only reported as open. If so lsof might not be accurate with
> threads ??
That's correct, lsof is walking the process list and counting open file
handles per process. It doesn't know, for example, that process 14678
and process 14679 are really two threads created from the same Java
process 14674.
Try something like this:
==
import java.io.*;
public class foo {
public static void main(String[] args) {
// Open one file
FileReader fr = new FileReader("foo.java");
while (true) {
// For every chracter typed in stdin ...
System.in.read();
// ... start a thread that does nothing but sleeping
new Thread() {
public void run() {
try {
Thread.sleep(6);
} catch (InterruptedException e) {
}
}
}.start();
}
}
}
==
In this program, you know you are opening only one file. Afterwards,
you are creating threads without doing anything. Run this program in
one window and watch the open file count of foo.java in another window.
You will see that the count goes up as the number of threads goes up.
This is by no means a Java issue. If you write a pthreads C program
similar to the above, you will observe the same thing.
> The reason we started looking at the open files is, we where having IO
> problems. After running for about 1 day, Linix started reporting IO errors
> and stating that it couldnt open any more files.
The problem might lie elsewhere. You can do a strace on the Java
process to figure out where and when files are opened.
>
> The other concern or question is, should the handle to a referenced jar file
> not be closed at some point?
My understanding is that each jar file is opened, mmap-ed and then
closed. So lsof will report the jar files to be open by the Java
process but that shouldn't take up a slot in the file handle table. (I
haven't looked at the source code of the JDK, so someone please correct
me if I'm wrong.)
--
Weiqi Gao
[EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Java Threads on Linux
At 2:16 PM +0200 10/3/01, Florent Coste wrote: >I'm not a kernel guru too, > >but what know is that : >java threads are mapped to posix threads (linuxthreads in the glibc). The >linuxthreads library uses the kernel threads, which directly map >into a 'process >like' entry in the scheduler. (1-1 mapping). One has already said >that the linux >kernel is optimized for the common case, ie few processus in the >scheduler queue. This isn't exactly accurate. Threads in linux (including Java threads) don't use posix threads, but the low, low level clone() system call. clone()ed processes, by default, share all their memory with each other, just like a thread. fork() builds off of clone() and implements copy-on-write between the processes. But that's not part of threading. clone()ed processes also share all the same file descriptors between themselves. I believe the only thing they do not share is their stack, obviously. That's what I remember from reading some article on it a while back, but feel free to correct me if I'm off somewhere. Avi -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Java 3D question
I have linux Java 3D working on my box. Are there any actual applications out there for Java 3D, or just the little demos that show that it works? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Java Threads on Linux
On Wed, Oct 03, 2001 at 06:50:48AM -0700, Avi Cherry wrote: > At 2:16 PM +0200 10/3/01, Florent Coste wrote: > >I'm not a kernel guru too, > > > >but what know is that : > >java threads are mapped to posix threads (linuxthreads in the glibc). The > >linuxthreads library uses the kernel threads, which directly map > >into a 'process > >like' entry in the scheduler. (1-1 mapping). One has already said > >that the linux > >kernel is optimized for the common case, ie few processus in the > >scheduler queue. > > > This isn't exactly accurate. Threads in linux (including Java > threads) don't use posix threads, but the low, low level clone() > system call. clone()ed processes, by default, share all their memory > with each other, just like a thread. fork() builds off of clone() > and implements copy-on-write between the processes. But that's not > part of threading. "POSIX threads" is an API, not a specification for implementation, and the JVM is using it. The current Linux implementation of POSIX threads uses the clone() kernel call for each thread, resulting in a 1-1 mapping to kernel threads. The work on a new POSIX threads implementation with Solaris-style NxM threads should vastly improve the Linux JVM's scalability. Nathan > > clone()ed processes also share all the same file descriptors between > themselves. I believe the only thing they do not share is their > stack, obviously. > > That's what I remember from reading some article on it a while back, > but feel free to correct me if I'm off somewhere. > > Avi > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > -- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Java Threads on Linux
On Wed, Oct 03, 2001 at 06:32:11PM -0700, Dan Kegel wrote: > root wrote: > > > > On Wed, Oct 03, 2001 at 02:22:51PM -0700, Dan Kegel wrote: > > > Nathan Meyers wrote: > > > > The current Linux implementation of POSIX threads uses the clone() kernel > > > > call for each thread, resulting in a 1-1 mapping to kernel threads. > > > > The work on a new POSIX threads implementation with Solaris-style NxM > > > > threads should vastly improve the Linux JVM's scalability. > > > > > > I dunno about that. clone() is pretty darn fast, and > > > there are patches to make the Linux scheduler handle > > > many running processes well. Once that scheduler > > > patch is in the standard kernel, there will be no real advantage > > > to NxM threads, IMHO. > > > > I'd be thrilled to see that happen. The problem has been, at least in > > part, exhaustion of precious resources (process IDs) - something that > > NxM threading models would help alleviate. Will the scheduler changes > > help with that? > > The 2.4 kernel uses 32 bit process ids, so that shouldn't be a > problem. Are there other precious resources you're worried about? > If not, there's a good chance NxM threading models will simply > introduce complexity rather than add performance. That was Linus' > bet long ago, and the question is still open. I'll defer to anyone who's faced this problem head-on. Any opinions from those who've hit the wall when scaling up Java on Linux? Nathan > > By the way, the scheduler patches are at > http://lse.sourceforge.net/scheduling/ > > - Dan > -- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
