Hi Ivo, I pulled the new client and ran it twice. First time it succeeded for 25 sessions and failed for 500 sessions, second time it failed for 25 sessions and exited. I'm attaching the logs, PFA.
Thanks Kusumakar On Sat, Apr 2, 2016 at 3:43 AM, Mihailovic Ivo <[email protected]> wrote: > Hi, > > > I have nothing to go on with this new error, so I modified the client to > print a stack trace on segfault. Please pull the client again, and if it > fails for you again, please send me the backtrace. > > > Best, > > Ivo > ------------------------------ > *From:* Mihailovic Ivo <[email protected]> > *Sent:* Thursday, March 31, 2016 11:33 AM > *To:* kusumakar bodha > > *Cc:* Neda Nasiriani; [email protected] > *Subject:* Re: Error media-streaming benchmark > > > Hi, > > > Thanks for testing the new build. Can you replicate the segfault? Does it > happen for any fixed argument, for example the same number of sessions? > Does it happen for all 4 clients or just one randomly? > > > Best, > > Ivo > ------------------------------ > *From:* kusumakar bodha <[email protected]> > *Sent:* Thursday, March 31, 2016 10:05 AM > *To:* Mihailovic Ivo > *Cc:* Neda Nasiriani; [email protected] > *Subject:* Re: Error media-streaming benchmark > > Hi Ivo, > > A new error showed up while running the benchmark: > > peak_hunter/launch_remote.sh: line 30: 221 Segmentation fault (core > dumped) httperf --hog --server 172.18.0.2 > --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] > --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log > --num-sessions=410 --rate=41 > > This error is not shown up in every run, it's coming sometimes. What could > be the issue? > > Thank you > Kusumakar > > On Thu, Mar 31, 2016 at 9:40 AM, kusumakar bodha <[email protected]> > wrote: > >> Hi Ivo, >> >> I pulled the new client and ran it, it is working fine now. My system has >> 72 cores, I think that was causing the trouble. Thank you very much for the >> support. >> >> Kusumakar >> >> On Thu, Mar 31, 2016 at 3:38 AM, Mihailovic Ivo <[email protected]> >> wrote: >> >>> Hi Neda and Bodha, >>> >>> >>> Can one of you try to pull the client again from dockerhub and run it on >>> the machine where it was failing? It should work now. >>> >>> >>> Ivo >>> ------------------------------ >>> *From:* Mihailovic Ivo <[email protected]> >>> *Sent:* Wednesday, March 30, 2016 3:05 PM >>> >>> *To:* Neda Nasiriani; kusumakar bodha >>> *Cc:* [email protected] >>> *Subject:* Re: Error media-streaming benchmark >>> >>> >>> Actually, just remove the -DHAVE_SCHED_AFFINITY part, and keep the >>> -DHAVE_EPOLL. >>> >>> >>> Ivo >>> ------------------------------ >>> *From:* Mihailovic Ivo <[email protected]> >>> *Sent:* Wednesday, March 30, 2016 2:41 PM >>> *To:* Neda Nasiriani; kusumakar bodha >>> *Cc:* [email protected] >>> *Subject:* Re: Error media-streaming benchmark >>> >>> >>> Hi, >>> >>> This is the code of the getaffinity system call: >>> >>> SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, >>> unsigned long __user *, user_mask_ptr) >>> { >>> int ret; >>> cpumask_var_t mask; >>> >>> if ((len * BITS_PER_BYTE) < nr_cpu_ids) >>> return -EINVAL; >>> if (len & (sizeof(unsigned long)-1)) >>> return -EINVAL; >>> >>> if (!alloc_cpumask_var(&mask, GFP_KERNEL)) >>> return -ENOMEM; >>> >>> ret = sched_getaffinity(pid, mask); >>> if (ret == 0) { >>> size_t retlen = min_t(size_t, len, cpumask_size()); >>> >>> if (copy_to_user(user_mask_ptr, mask, retlen)) >>> ret = -EFAULT; >>> else >>> ret = retlen; >>> } >>> free_cpumask_var(mask); >>> >>> return ret; >>> } >>> >>> >>> Given that videoperf passes sizeof(long) for the len parameter, this is >>> going to fail if you run it on a machine with more than 64 cores (the first >>> if statement in the code). Do you have more than 64 cores on your machine? >>> >>> Anyway, the simplest workaround for the problem is: >>> >>> 1) Get the sources from: >>> >>> https://github.com/ParsaLab/cloudsuite/tree/master/benchmarks/media-streaming >>> >>> 2) Comment out this line in client/files/videoperf/Makefile.in: >>> DEFS += -DHAVE_SCHED_AFFINITY -DHAVE_EPOLL >>> >>> 3) Build the client yourself: >>> docker build --rm -t cloudsuite/media-streaming:client /path/to/client >>> >>> Please let me know if this works for you. >>> >>> Best, >>> Ivo >>> ------------------------------ >>> *From:* Neda Nasiriani <[email protected]> >>> *Sent:* Wednesday, March 30, 2016 1:40 PM >>> *To:* kusumakar bodha >>> *Cc:* [email protected] >>> *Subject:* Re: Error media-streaming benchmark >>> >>> Hello kusumakar, >>> >>> I have the same problem and it happens on machines with more than 4 >>> cores. If you try it on a machine with 4 or less cores this does not >>> happen. It could be related to the cpu mask which is creating this problem >>> for machines with >4 cores. I hope it get resolved. Please let me know if >>> you have any findings. Thanks >>> >>> On Wednesday, March 30, 2016, kusumakar bodha <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> >>>> I'm trying to run the cloudsuite media-streaming benchmark and >>>> encountered the following error after I started the client. >>>> >>>> >>>> >>>> Total clients = 4 >>>> Minimum number of sessions = 25 >>>> Maximum number of sessions = 500 >>>> Launching 4 clients on localhost >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log >>>> --num-sessions=25 --rate=2 >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log >>>> --num-sessions=25 --rate=2 >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log >>>> --num-sessions=25 --rate=2 >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log >>>> --num-sessions=25 --rate=2 >>>> sizeof(fd_set) = 128 >>>> sizeof(fd_set) = 128 >>>> sizeof(fd_set) = 128 >>>> sizeof(fd_set) = 128 >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= >>>> Run on hostname: f2ef7e2c3e74 >>>> Run on hostname: f2ef7e2c3e74 >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= >>>> Run on hostname: f2ef7e2c3e74 >>>> Run at: Wed Mar 30 05:02:02 2016 >>>> Run on hostname: f2ef7e2c3e74 >>>> Run at: Wed Mar 30 05:02:02 2016 >>>> >>>> >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> Run at: Wed Mar 30 05:02:02 2016 >>>> >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> Run at: Wed Mar 30 05:02:02 2016 >>>> >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> grep: /output/*: No such file or directory >>>> grep: /output/*: No such file or directory >>>> peak_hunter/launch_hunt_bin.sh: line 56: 0*100/0: division by 0 (error >>>> token is "0") >>>> Benchmark succeeded for 25 sessions >>>> Launching 4 clients on localhost >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log >>>> --num-sessions=500 --rate=50 >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log >>>> --num-sessions=500 --rate=50 >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log >>>> --num-sessions=500 --rate=50 >>>> Running command httperf --hog --server streaming_server >>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log >>>> --num-sessions=500 --rate=50 >>>> sizeof(fd_set) = 128 >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>> --call-stats-file= >>>> Run on hostname: f2ef7e2c3e74 >>>> sizeof(fd_set) = 128 >>>> Run at: Wed Mar 30 05:02:02 2016 >>>> >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>> --call-stats-file= >>>> Run on hostname: f2ef7e2c3e74 >>>> Run at: Wed Mar 30 05:02:02 2016 >>>> >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> sizeof(fd_set) = 128 >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>> --call-stats-file= >>>> Run on hostname: f2ef7e2c3e74 >>>> Run at: Wed Mar 30 05:02:03 2016 >>>> >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> sizeof(fd_set) = 128 >>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>> --call-stats-file= >>>> Run on hostname: f2ef7e2c3e74 >>>> Run at: Wed Mar 30 05:02:03 2016 >>>> >>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>> grep: /output/*: No such file or directory >>>> grep: /output/*: No such file or directory >>>> peak_hunter/launch_hunt_bin.sh: line 56: 0*100/0: division by 0 (error >>>> token is "0") >>>> Benchmark succeeded for 500 sessions >>>> Maximum limit for number of sessions too low. >>>> grep: /output/*: No such file or directory >>>> (standard_in) 2: syntax error >>>> grep: /output/*: No such file or directory >>>> (standard_in) 2: syntax error >>>> grep: /output/*: No such file or directory >>>> (standard_in) 2: syntax error >>>> grep: /output/*: No such file or directory >>>> (standard_in) 2: syntax error >>>> grep: /output/*: No such file or directory >>>> (standard_in) 2: syntax error >>>> Requests: >>>> Replies: >>>> (standard_in) 1: syntax error >>>> Reply rate: >>>> (standard_in) 1: syntax error >>>> Reply time: >>>> Net I/O: >>>> >>>> What is the problem? The client is exiting immediately >>>> I'm using Docker version 1.9.0, build 76d6bc9 and kernel: >>>> 4.4.0-14-generic >>>> on ubuntu 14.04 >>>> >>>> Thank you >>>> Kusumakar >>>> >>> >>> >>> -- >>> Best Regards, >>> Neda >>> >> >> >
firstRunMediaStreaming.log
Description: Binary data
secondRunMediaStreaming.log
Description: Binary data
