Hi Ivo, Additional information: I'm actually trying to gather some performance characteristics of the processor when only the server is running, so I pinned each of the client and server to one of the cores (different cores) and running the benchmark.
Thank you, Kusumakar On Mon, Apr 4, 2016 at 9:44 AM, kusumakar bodha <[email protected]> wrote: > Hi Ivo, > > I pulled the new client and ran it twice. First time it succeeded for 25 > sessions and failed for 500 sessions, second time it failed for 25 sessions > and exited. I'm attaching the logs, PFA. > > Thanks > Kusumakar > > On Sat, Apr 2, 2016 at 3:43 AM, Mihailovic Ivo <[email protected]> > wrote: > >> Hi, >> >> >> I have nothing to go on with this new error, so I modified the client to >> print a stack trace on segfault. Please pull the client again, and if it >> fails for you again, please send me the backtrace. >> >> >> Best, >> >> Ivo >> ------------------------------ >> *From:* Mihailovic Ivo <[email protected]> >> *Sent:* Thursday, March 31, 2016 11:33 AM >> *To:* kusumakar bodha >> >> *Cc:* Neda Nasiriani; [email protected] >> *Subject:* Re: Error media-streaming benchmark >> >> >> Hi, >> >> >> Thanks for testing the new build. Can you replicate the segfault? Does it >> happen for any fixed argument, for example the same number of sessions? >> Does it happen for all 4 clients or just one randomly? >> >> >> Best, >> >> Ivo >> ------------------------------ >> *From:* kusumakar bodha <[email protected]> >> *Sent:* Thursday, March 31, 2016 10:05 AM >> *To:* Mihailovic Ivo >> *Cc:* Neda Nasiriani; [email protected] >> *Subject:* Re: Error media-streaming benchmark >> >> Hi Ivo, >> >> A new error showed up while running the benchmark: >> >> peak_hunter/launch_remote.sh: line 30: 221 Segmentation fault >> (core dumped) httperf --hog --server 172.18.0.2 >> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log >> --num-sessions=410 --rate=41 >> >> This error is not shown up in every run, it's coming sometimes. What >> could be the issue? >> >> Thank you >> Kusumakar >> >> On Thu, Mar 31, 2016 at 9:40 AM, kusumakar bodha <[email protected] >> > wrote: >> >>> Hi Ivo, >>> >>> I pulled the new client and ran it, it is working fine now. My system >>> has 72 cores, I think that was causing the trouble. Thank you very much for >>> the support. >>> >>> Kusumakar >>> >>> On Thu, Mar 31, 2016 at 3:38 AM, Mihailovic Ivo <[email protected]> >>> wrote: >>> >>>> Hi Neda and Bodha, >>>> >>>> >>>> Can one of you try to pull the client again from dockerhub and run it >>>> on the machine where it was failing? It should work now. >>>> >>>> >>>> Ivo >>>> ------------------------------ >>>> *From:* Mihailovic Ivo <[email protected]> >>>> *Sent:* Wednesday, March 30, 2016 3:05 PM >>>> >>>> *To:* Neda Nasiriani; kusumakar bodha >>>> *Cc:* [email protected] >>>> *Subject:* Re: Error media-streaming benchmark >>>> >>>> >>>> Actually, just remove the -DHAVE_SCHED_AFFINITY part, and keep the >>>> -DHAVE_EPOLL. >>>> >>>> >>>> Ivo >>>> ------------------------------ >>>> *From:* Mihailovic Ivo <[email protected]> >>>> *Sent:* Wednesday, March 30, 2016 2:41 PM >>>> *To:* Neda Nasiriani; kusumakar bodha >>>> *Cc:* [email protected] >>>> *Subject:* Re: Error media-streaming benchmark >>>> >>>> >>>> Hi, >>>> >>>> This is the code of the getaffinity system call: >>>> >>>> SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, >>>> unsigned long __user *, user_mask_ptr) >>>> { >>>> int ret; >>>> cpumask_var_t mask; >>>> >>>> if ((len * BITS_PER_BYTE) < nr_cpu_ids) >>>> return -EINVAL; >>>> if (len & (sizeof(unsigned long)-1)) >>>> return -EINVAL; >>>> >>>> if (!alloc_cpumask_var(&mask, GFP_KERNEL)) >>>> return -ENOMEM; >>>> >>>> ret = sched_getaffinity(pid, mask); >>>> if (ret == 0) { >>>> size_t retlen = min_t(size_t, len, cpumask_size()); >>>> >>>> if (copy_to_user(user_mask_ptr, mask, retlen)) >>>> ret = -EFAULT; >>>> else >>>> ret = retlen; >>>> } >>>> free_cpumask_var(mask); >>>> >>>> return ret; >>>> } >>>> >>>> >>>> Given that videoperf passes sizeof(long) for the len parameter, this is >>>> going to fail if you run it on a machine with more than 64 cores (the first >>>> if statement in the code). Do you have more than 64 cores on your machine? >>>> >>>> Anyway, the simplest workaround for the problem is: >>>> >>>> 1) Get the sources from: >>>> >>>> https://github.com/ParsaLab/cloudsuite/tree/master/benchmarks/media-streaming >>>> >>>> 2) Comment out this line in client/files/videoperf/Makefile.in: >>>> DEFS += -DHAVE_SCHED_AFFINITY -DHAVE_EPOLL >>>> >>>> 3) Build the client yourself: >>>> docker build --rm -t cloudsuite/media-streaming:client /path/to/client >>>> >>>> Please let me know if this works for you. >>>> >>>> Best, >>>> Ivo >>>> ------------------------------ >>>> *From:* Neda Nasiriani <[email protected]> >>>> *Sent:* Wednesday, March 30, 2016 1:40 PM >>>> *To:* kusumakar bodha >>>> *Cc:* [email protected] >>>> *Subject:* Re: Error media-streaming benchmark >>>> >>>> Hello kusumakar, >>>> >>>> I have the same problem and it happens on machines with more than 4 >>>> cores. If you try it on a machine with 4 or less cores this does not >>>> happen. It could be related to the cpu mask which is creating this problem >>>> for machines with >4 cores. I hope it get resolved. Please let me know if >>>> you have any findings. Thanks >>>> >>>> On Wednesday, March 30, 2016, kusumakar bodha <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> >>>>> I'm trying to run the cloudsuite media-streaming benchmark and >>>>> encountered the following error after I started the client. >>>>> >>>>> >>>>> >>>>> Total clients = 4 >>>>> Minimum number of sessions = 25 >>>>> Maximum number of sessions = 500 >>>>> Launching 4 clients on localhost >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log >>>>> --num-sessions=25 --rate=2 >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log >>>>> --num-sessions=25 --rate=2 >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log >>>>> --num-sessions=25 --rate=2 >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log >>>>> --num-sessions=25 --rate=2 >>>>> sizeof(fd_set) = 128 >>>>> sizeof(fd_set) = 128 >>>>> sizeof(fd_set) = 128 >>>>> sizeof(fd_set) = 128 >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> Run on hostname: f2ef7e2c3e74 >>>>> Run on hostname: f2ef7e2c3e74 >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> Run on hostname: f2ef7e2c3e74 >>>>> Run at: Wed Mar 30 05:02:02 2016 >>>>> Run on hostname: f2ef7e2c3e74 >>>>> Run at: Wed Mar 30 05:02:02 2016 >>>>> >>>>> >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> Run at: Wed Mar 30 05:02:02 2016 >>>>> >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> Run at: Wed Mar 30 05:02:02 2016 >>>>> >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> grep: /output/*: No such file or directory >>>>> grep: /output/*: No such file or directory >>>>> peak_hunter/launch_hunt_bin.sh: line 56: 0*100/0: division by 0 (error >>>>> token is "0") >>>>> Benchmark succeeded for 25 sessions >>>>> Launching 4 clients on localhost >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log >>>>> --num-sessions=500 --rate=50 >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log >>>>> --num-sessions=500 --rate=50 >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log >>>>> --num-sessions=500 --rate=50 >>>>> Running command httperf --hog --server streaming_server >>>>> --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] >>>>> --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log >>>>> --num-sessions=500 --rate=50 >>>>> sizeof(fd_set) = 128 >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> Run on hostname: f2ef7e2c3e74 >>>>> sizeof(fd_set) = 128 >>>>> Run at: Wed Mar 30 05:02:02 2016 >>>>> >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> Run on hostname: f2ef7e2c3e74 >>>>> Run at: Wed Mar 30 05:02:02 2016 >>>>> >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> sizeof(fd_set) = 128 >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> Run on hostname: f2ef7e2c3e74 >>>>> Run at: Wed Mar 30 05:02:03 2016 >>>>> >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> sizeof(fd_set) = 128 >>>>> httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ >>>>> --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll >>>>> --call-stats-file= >>>>> Run on hostname: f2ef7e2c3e74 >>>>> Run at: Wed Mar 30 05:02:03 2016 >>>>> >>>>> httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) >>>>> grep: /output/*: No such file or directory >>>>> grep: /output/*: No such file or directory >>>>> peak_hunter/launch_hunt_bin.sh: line 56: 0*100/0: division by 0 (error >>>>> token is "0") >>>>> Benchmark succeeded for 500 sessions >>>>> Maximum limit for number of sessions too low. >>>>> grep: /output/*: No such file or directory >>>>> (standard_in) 2: syntax error >>>>> grep: /output/*: No such file or directory >>>>> (standard_in) 2: syntax error >>>>> grep: /output/*: No such file or directory >>>>> (standard_in) 2: syntax error >>>>> grep: /output/*: No such file or directory >>>>> (standard_in) 2: syntax error >>>>> grep: /output/*: No such file or directory >>>>> (standard_in) 2: syntax error >>>>> Requests: >>>>> Replies: >>>>> (standard_in) 1: syntax error >>>>> Reply rate: >>>>> (standard_in) 1: syntax error >>>>> Reply time: >>>>> Net I/O: >>>>> >>>>> What is the problem? The client is exiting immediately >>>>> I'm using Docker version 1.9.0, build 76d6bc9 and kernel: >>>>> 4.4.0-14-generic >>>>> on ubuntu 14.04 >>>>> >>>>> Thank you >>>>> Kusumakar >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Neda >>>> >>> >>> >> >
