max user processes (-u) 47555 is the limit that is probably being hit. So you probably have 47555+ threads in execution on that host. I honestly don't know where all of the threads are coming from. Storm will create two threads for each executor, and a small fixed number more for the system in each worker. By increasing the parallelism hint of your bolts/spouts and increasing the number of workers the number of threads will go up and linux treats threads against your max user processes limit. Most of the time when I run into a situation like this I try to get a jstack or a heap dump of the workers to try and understand what all of the threads are associated with. My guess is that something is happening with HBase, like you said, where the client is not shutting down all of it's threads when you close the table. - Bobby
On Thursday, May 12, 2016 9:00 AM, numan goceri <numangoc...@yahoo.com> wrote: Hi Bobby, Thanks for your reply. I've searched a lot on the net, what OutOfMemoryError actually means. And it seems like an OS resource problem.Link#1: http://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-threadLink#2: https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread I'm running this on my VM-Player and so is the ulimit configurations:[root@my_Project]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 47555 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 47555 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@my_Project]# I'm guessing Storm is not killing any threads, which have been opened so far. if it's opening a new thread, each time when I want to "put" a result into the hbase, then it should kill, since I call "close" afterwards. My code is basically like followings: prepare(){ this.connection = ConnectionFactory.createConnection(constructConfiguration()); this.allTuplesTable = connection.getTable("allTuples"); this.resultsTable = connection.getTable("results"); } execute(){ readFromAllTuples(); ... this.resultsTable.put(put); this.resultsTable.close();}cleanup(){ this.allTuplesTable.close(); this.resultsTable.close();} I'd appreciate if you could show me a way to continue.. the data, I'm dealing with, should be not soo "big data" for Storm. --- Numan Göceri On Thursday, May 12, 2016 3:31 PM, Bobby Evans <ev...@yahoo-inc.com> wrote: Numan, Please read up a bit on what the OutOfMemeroyError actually means. https://duckduckgo.com/?q=java.lang.OutOfMemoryError%3A+unable+to+create+new+native+thread&t=ffab Most of the time on Linux it means that you have hit a ulimit on the number of processes/threads that your headless user can handle. By increasing the parallelism you did not make the problem go away, you just created a new problem, and when you fixed that problem the original problem came back. You either need to increase the ulimit (possibly both hard and soft) for the user your worker processes are running as, or you need to find a way to reduce the number of threads that you are using. - Bobby On Thursday, May 12, 2016 8:07 AM, numan goceri <numangoc...@yahoo.com.INVALID> wrote: I have implemented a topology by using 1 Spout and 2 Bolts: |Spout|->|Bolt1|->|Bolt2| Kafka producer pushes the input rows out of a csv file (around 2000 rows) currently and Spout is receiving them all in once. Bolt1: writes all incoming tuples into an Hbase Table (htable_allTuples) Bolt2: checks all incoming tuples and once the expected tuple arrived, then it reads other related tuples from "htable_allTuples" and writes the results into another hbase table (htable_result) In my Bolt2, if the conditions are so many that finally I have 2 rows to write into the result table, then it all works very fine. But if in my Bolt2, if I reduce the conditions, so that there will be more than 2 rows as a result (like around 18 rows), then my Bolt2 throws the following error: "java.lang.RuntimeException: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor.." The solution for this problem as I've realized is parallelism: using more workers, executors and/or tasks. When I increase the number of executors and workers, then I receive the following hbase connection error: "ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 4 attempts." "baseZNode=/hbase-unsecure Unable to set watcher on znode (/hbase-unsecure/hbaseid)" So I found out that I should increase the value of "maxClientCnxns" parameter. (by default: 300). I've first set it to 3000 and I still receive the same error and then I've set to 0, which means: no client connection limit between hbase and zookeeper. This time I receive again my old error message: "java.lang.OutOfMemoryError: unable to create new native thread". I open the hbase table connections once in "prepare" method and close them all in "cleanup" method. Whenever I call <hbaseTable>.put(..) method, I call also <hbaseTable>.close(); but still it seems like there are lots of threads runnning in background. Do you have any idea how to get rid of these two problems? and how to set a clean topology? Thanks in advance for the feedbacks. --- Numan Göceri