I tested another topology and worked fine me . But Thanks for your time. I hope can someone figure where is the problem
On Sunday, July 2, 2017, J.R. Pauley <[email protected]> wrote: > Sam > seems like you don't really know memory is the issue. Have you had ANY drpc working in your environemtn? If not maybe best to start with a minimal test implementation just to confirm or eliminate memory as an issue. Sorry I don't have better idea. > On Sun, Jul 2, 2017 at 2:13 AM, sam mohel <[email protected]> wrote: >> >> is there any help , i tried to create swap file to increase the size but didn't fix the problem . is there any other way i can try it ? >> On Sat, Jul 1, 2017 at 4:50 PM, sam mohel <[email protected]> wrote: >>> >>> I don't have any errors in log files except drpc.log i have this >>> 2017-07-01T07:34:02.660+0200 b.s.d.drpc [INFO] Starting Distributed RPC servers... >>> 2017-07-01T07:34:47.267+0200 o.a.t.s.TNonblockingServer [WARN] Got an IOException in internalRead! >>> java.io.IOException: Connection reset by peer >>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_121] >>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_121] >>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_121] >>> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_121] >>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384) ~[na:1.7.0_121] >>> at org.apache.thrift7.transport.TNonblockingSocket.read(TNonblockingSocket.java:141) ~[storm-core-0.9.6.jar:0.9.6] >>> at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.internalRead(TNonblockingServer.java:669) [storm-core-0.9.6.jar:0.9.6] >>> at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:458) [storm-core-0.9.6.jar:0.9.6] >>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359) [storm-core-0.9.6.jar:0.9.6] >>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304) [storm-core-0.9.6.jar:0.9.6] >>> at org.apache.thrift7.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243) [storm-core-0.9.6.jar:0.9.6] >>> 2017-07-01T07:44:17.639+0200 b.s.d.drpc [WARN] Timeout DRPC request id: 1 start at 1498887255 >>> and storm ui have numbers in their columns except transferred , capacity and process latency . >>> i faced something like that before with a topology and when i increased RAM the problem fixed , i think that is the reason here also with his topology but i couldn't increase the RAM again >>> On Sat, Jul 1, 2017 at 11:09 AM, J.R. Pauley <[email protected]> wrote: >>>> >>>> Sam >>>> I would guess timeout is not your problem as defaults seem generous. The little I've learned the last couple days tell me to look at drpc.log and topology logs for some issue. Also I learned from Bobby I can't test this sort of thing in a LocalDRPC mode even with drpc client on same box it won't work, has to be a regular submitted topology. Other thing I found was drpc client version may matter. I reverted to 0.9.6 because everything was easier to configure. In 1.0.2 the client you have to give a config param map with lots of params that have to match the server config, and that sounds like an easy place to have some mismatch >>>> On Fri, Jun 30, 2017 at 6:13 PM, sam mohel <[email protected]> wrote: >>>>> >>>>> I searched and found this .. "The timeout on DRPC requests within the DRPC server. Defaults to 10 minutes. Note that requests can also timeout based on the socket timeout on the DRPC client, and separately based on the topology message timeout for the topology implementing the DRPC function." >>>>> >>>>> Should i change drpc.request.timeout.secs only or what ? >>>>> Any help i will appreciate >>>>> On Fri, Jun 30, 2017 at 7:16 PM, sam mohel <[email protected]> wrote: >>>>>> >>>>>> sorry the full error i got >>>>>> Exception in thread "main" DRPCExecutionException(msg:Request timed out) >>>>>> at backtype.storm.generated.DistributedRPC$execute_result.read(DistributedRPC.java:904) >>>>>> at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78) >>>>>> at backtype.storm.generated.DistributedRPC$Client.recv_execute(DistributedRPC.java:92) >>>>>> at backtype.storm.generated.DistributedRPC$Client.execute(DistributedRPC.java:78) >>>>>> at backtype.storm.utils.DRPCClient.execute(DRPCClient.java:71) >>>>>> >>>>>> On Fri, Jun 30, 2017 at 3:36 PM, sam mohel <[email protected]> wrote: >>>>>>> >>>>>>> @jim yes I ran ./storm drpc in machine A with nimbus . And got the error >>>>>>> >>>>>>> DRPCExecutionException(msg:Request failed) >>>>>>> >>>>>>> in console after sometimes from submitting the topology >>>>>>> >>>>>>> On Friday, June 30, 2017, Pauley, Jim <[email protected]> wrote: >>>>>>> > Sam >>>>>>> > Re Bobby comment that your DRPC was not even running, did you manually start the drpc from console and get it running? I know you can see if it is listening with netstat -ant | grep 3772 >>>>>>> > ________________________________ >>>>>>> > From: sam mohel [[email protected]] >>>>>>> > Sent: Thursday, June 29, 2017 9:45 PM >>>>>>> > To: [email protected] >>>>>>> > Subject: [EXTERNAL] Re: DRPC problem >>>>>>> > >>>>>>> > excuse me . i tried to submit another topology and worked well with same configurations . So why this topology has a problem !! How can i figure where is the problem ? >>>>>>> > Any help i'll really appreciate it >>>>>>> > On Thu, Jun 29, 2017 at 8:35 PM, sam mohel <[email protected]> wrote: >>>>>>> >> >>>>>>> >> Thanks for replying , really i will loss my mind from this error . Can i change ip address or what should i do with Drpc server ? >>>>>>> >> On Thu, Jun 29, 2017 at 8:29 PM, Bobby Evans <[email protected]> wrote: >>>>>>> >>> >>>>>>> >>> Your error message shows that your DRPC server isn't even running, so you don't need to worry about the amount of memory until you actually have the processes up and running. >>>>>>> >>> Once you have them running then you can look at the GC metrics to see if it looks like you need to give it more heap. This is generic for java in general and has little to do with storm in particular. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> - Bobby >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> On Thursday, June 29, 2017, 1:18:36 PM CDT, sam mohel < [email protected]> wrote: >>>>>>> >>> >>>>>>> >>> Thanks but how can i know if Memory is suitable for my project or not ? >>>>>>> >>> On Thu, Jun 29, 2017 at 8:10 PM, Bobby Evans < [email protected]> wrote: >>>>>>> >>> >>>>>>> >>> No it has nothing to do with GC. It means that the command line confused the JVM and it thought you wanted to run a the main method in a class called _JAAS_PLACEHOLDER. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> - Bobby >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> On Thursday, June 29, 2017, 12:45:16 PM CDT, sam mohel < [email protected]> wrote: >>>>>>> >>> >>>>>>> >>> Excuse me . Is that line >>>>>>> >>> >>>>>>> >>> Drpc.childopts"-Xmx768m _JAAS_PLACEHOLDER -Xloggc:/var/log/storm/drpc- gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps" >>>>>>> >>> >>>>>>> >>> Means to check if there's a problem with garbage collector or not ? >>>>>>> >>> And about want I got >>>>>>> >>> >>>>>>> >>> Could not find or load main class _JAAS_PLACEHOLDER . Is that means error I should fix and I have problem with GC ? >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> > Delete _JAAS_PLACEHOLDER. It is there as something you should replace if you want to have security for your DRPC server. >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > - Bobby >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > On Thursday, June 29, 2017, 10:25:24 AM CDT, sam mohel < [email protected]> wrote: >>>>>>> >>> > >>>>>>> >>> > Really appreciate your time Bobby . So the error I got when I added this line to drpc.childopts >>>>>>> >>> > "-Xmx768m _JAAS_PLACEHOLDER -Xloggc:/var/log/storm/drpc- gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps" >>>>>>> >>> > I got Error: Could not find or load main class _JAAS_PLACEHOLDER how can I fix it ? >>>>>>> >>> > >>>>>>> >>> > On Thursday, June 29, 2017, Bobby Evans <[email protected]> wrote: >>>>>>> >>> >> I don't see issues with your configs on the surface. >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> >> - Bobby >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> >> On Thursday, June 29, 2017, 10:02:37 AM CDT, sam mohel < [email protected]> wrote: >>>>>>> >>> >> >>>>>>> >>> >> Are my configurations right ? >>>>>>> >>> >> I got in terminal when submitted the topology after some times >>>>>>> >>> >> drpcexecutionexception(msg: request timed out) >>>>>>> >>> >> In log file of drpc I got >>>>>>> >>> >> [INFO] Starting Distributed RPC servers... >>>>>>> >>> >> [WARN] Timeout DRPC request id: 200 start at 1498657923 >>>>>>> >>> >> In worker log file doesn't receive any error message . >>>>>>> >>> >> But I tried to add this drpc.childopts: "-Xmx768m _JAAS_PLACEHOLDER -Xloggc:/var/log/storm/drpc- gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps" in storm.yaml to check if there is problem in RAM and retsart submitting again . >>>>>>> >>> >> I got in drpc terminal >>>>>>> >>> >> Error: Could not find or load main class _JAAS_PLACEHOLDER >>>>>>> >>> >> And in terminal when submitted >>>>>>> >>> >> Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport. TTransportException: java.net.ConnectException: Connection refused (Connection refused) >>>>>>> >>> >> at backtype.storm.utils. DRPCClient.<init>(DRPCClient. java:42) >>>>>>> >>> >> at backtype.storm.utils. DRPCClient.<init>(DRPCClient. java:47) >>>>>>> >>> >> at trident.FirstStoryDetection. main(FirstStoryDetection.java: 308) >>>>>>> >>> >> Caused by: org.apache.thrift7.transport. TTransportException: java.net.ConnectException: Connection refused (Connection refused) >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> >> On Thursday, June 29, 2017, Bobby Evans <[email protected]> wrote: >>>>>>> >>> >>> You are going to need to look at the logs for your topology and the logs for the drpc server to see if there is anything in there that indicates what is happening. >>>>>>> >>> >>> >>>>>>> >>> >>> >>>>>>> >>> >>> - Bobby >>>>>>> >>> >>> >>>>>>> >>> >>> >>>>>>> >>> >>> On Wednesday, June 28, 2017, 11:21:14 PM CDT, sam mohel < [email protected]> wrote: >>>>>>> >>> >>> >>>>>>> >>> >>> I submitted topology in local without any problem , but in production mode i couldn't as you can see in ui zeros values in columns except execute columns . >>>>>>> >>> >>> i got after sometimes in terminal drpcexecutionexception(msg: request timed out) >>>>>>> >>> >>> my configurations are >>>>>>> >>> >>> Machine A and Machine B >>>>>>> >>> >>> storm.yaml in Machine A is >>>>>>> >>> >>> storm.zookeeper.servers: >>>>>>> >>> >>> - "192.168.x.x" >>>>>>> >>> >>> >>>>>>> >>> >>> nimbus.host : "192.168.x.x" >>>>>>> >>> >>> supervisor.childopts: "-Xmx4g" >>>>>>> >>> >>> worker.childopts: "-Xmx4g" >>>>>>> >>> >>> storm.yaml in Machine B is >>>>>>> >>> >>> storm.zookeeper.servers: >>>>>>> >>> >>> - "192.168.x.x" >>>>>>> >>> >>> >>>>>>> >>> >>> nimbus.host : "192.168.x.x" >>>>>>> >>> >>> supervisor.childopts: "-Xmx4g" >>>>>>> >>> >>> worker.childopts: "-Xmx4g" >>>>>>> >>> >>> i set drpc in the code >>>>>>> >>> >>> Config conf = new Config(); >>>>>>> >>> >>> List<String> dprcServers = new ArrayList<String>(); >>>>>>> >>> >>> dprcServers.add("192.168.x.x") ; >>>>>>> >>> >>> conf.put(Config.DRPC_SERVERS, dprcServers); >>>>>>> >>> >>> conf.put(Config.DRPC_PORT, 3772); >>>>>>> >>> >>> // distributed mode >>>>>>> >>> >>> Config conf = createTopologyConfiguration( prop, true); >>>>>>> >>> >>> LocalDRPC drpc = null; >>>>>>> >>> >>> StormSubmitter.submitTopology( args[0], conf, buildTopology(drpc)); >>>>>>> >>> >>> client=new DRPCClient("192.168.x.x", 3772); >>>>>>> >>> >>> i used same ip address for storm.zookeeper.servers , nimbus.host ,dprcServers and DRPCClient . Is that wrong ? >>>>>>> >>> >>> and i ran nimbus , drpc,ui in Machine A , I ran supervisor in Machine B >>>>>>> >>> >>> i appreciate any help , Thanks >>>>>>> >>> >>> >>>>>>> >>> >>> >>>>>>> >> >>>>>>> > >>>>>>> > >>>>> >>>> >>> >> > >
