Hi,
It is supposed to fail for 500 sessions per client. The way it works is - it does a binary search of the number of sessions/connections for which the error percentage is less than 5%. This is the peak throughput with the acceptable error rate and is reported in the end. The output is quite a bit noisy, but you should have the result logs in the output folder at the end. The second run is not good, because connections are failing with only 25 sessions per client. But to know what happened, we need to look into the result files which lists errors by type. And, of course, the segfaults should not be happening. It's really weird that this bug happens non-deterministically, given that videoperf is a single-threaded application. I tried looking at the code, but can't figure it out, the code seems good. It's as if the client is getting a response when it's not expecting it, i.e. when it didn't send a request. I'll to figure it out, but so far I haven't been able to replicate it. You can also try to debug it yourself if you know your way around gdb. Best, Ivo ________________________________ From: kusumakar bodha <[email protected]> Sent: Monday, April 4, 2016 9:47 PM To: Mihailovic Ivo Cc: Neda Nasiriani; [email protected] Subject: Re: Error media-streaming benchmark Hi Ivo, Additional information: I'm actually trying to gather some performance characteristics of the processor when only the server is running, so I pinned each of the client and server to one of the cores (different cores) and running the benchmark. Thank you, Kusumakar On Mon, Apr 4, 2016 at 9:44 AM, kusumakar bodha <[email protected]<mailto:[email protected]>> wrote: Hi Ivo, I pulled the new client and ran it twice. First time it succeeded for 25 sessions and failed for 500 sessions, second time it failed for 25 sessions and exited. I'm attaching the logs, PFA. Thanks Kusumakar On Sat, Apr 2, 2016 at 3:43 AM, Mihailovic Ivo <[email protected]<mailto:[email protected]>> wrote: Hi, I have nothing to go on with this new error, so I modified the client to print a stack trace on segfault. Please pull the client again, and if it fails for you again, please send me the backtrace. Best, Ivo ________________________________ From: Mihailovic Ivo <[email protected]<mailto:[email protected]>> Sent: Thursday, March 31, 2016 11:33 AM To: kusumakar bodha Cc: Neda Nasiriani; [email protected]<mailto:[email protected]> Subject: Re: Error media-streaming benchmark Hi, Thanks for testing the new build. Can you replicate the segfault? Does it happen for any fixed argument, for example the same number of sessions? Does it happen for all 4 clients or just one randomly? Best, Ivo ________________________________ From: kusumakar bodha <[email protected]<mailto:[email protected]>> Sent: Thursday, March 31, 2016 10:05 AM To: Mihailovic Ivo Cc: Neda Nasiriani; [email protected]<mailto:[email protected]> Subject: Re: Error media-streaming benchmark Hi Ivo, A new error showed up while running the benchmark: peak_hunter/launch_remote.sh: line 30: 221 Segmentation fault (core dumped) httperf --hog --server 172.18.0.2 --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log --num-sessions=410 --rate=41 This error is not shown up in every run, it's coming sometimes. What could be the issue? Thank you Kusumakar On Thu, Mar 31, 2016 at 9:40 AM, kusumakar bodha <[email protected]<mailto:[email protected]>> wrote: Hi Ivo, I pulled the new client and ran it, it is working fine now. My system has 72 cores, I think that was causing the trouble. Thank you very much for the support. Kusumakar On Thu, Mar 31, 2016 at 3:38 AM, Mihailovic Ivo <[email protected]<mailto:[email protected]>> wrote: Hi Neda and Bodha, Can one of you try to pull the client again from dockerhub and run it on the machine where it was failing? It should work now. Ivo ________________________________ From: Mihailovic Ivo <[email protected]<mailto:[email protected]>> Sent: Wednesday, March 30, 2016 3:05 PM To: Neda Nasiriani; kusumakar bodha Cc: [email protected]<mailto:[email protected]> Subject: Re: Error media-streaming benchmark Actually, just remove the -DHAVE_SCHED_AFFINITY part, and keep the -DHAVE_EPOLL. Ivo ________________________________ From: Mihailovic Ivo <[email protected]<mailto:[email protected]>> Sent: Wednesday, March 30, 2016 2:41 PM To: Neda Nasiriani; kusumakar bodha Cc: [email protected]<mailto:[email protected]> Subject: Re: Error media-streaming benchmark Hi, This is the code of the getaffinity system call: SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, unsigned long __user *, user_mask_ptr) { int ret; cpumask_var_t mask; if ((len * BITS_PER_BYTE) < nr_cpu_ids) return -EINVAL; if (len & (sizeof(unsigned long)-1)) return -EINVAL; if (!alloc_cpumask_var(&mask, GFP_KERNEL)) return -ENOMEM; ret = sched_getaffinity(pid, mask); if (ret == 0) { size_t retlen = min_t(size_t, len, cpumask_size()); if (copy_to_user(user_mask_ptr, mask, retlen)) ret = -EFAULT; else ret = retlen; } free_cpumask_var(mask); return ret; } Given that videoperf passes sizeof(long) for the len parameter, this is going to fail if you run it on a machine with more than 64 cores (the first if statement in the code). Do you have more than 64 cores on your machine? Anyway, the simplest workaround for the problem is: 1) Get the sources from: https://github.com/ParsaLab/cloudsuite/tree/master/benchmarks/media-streaming 2) Comment out this line in client/files/videoperf/Makefile.in: DEFS += -DHAVE_SCHED_AFFINITY -DHAVE_EPOLL 3) Build the client yourself: docker build --rm -t cloudsuite/media-streaming:client /path/to/client Please let me know if this works for you. Best, Ivo ________________________________ From: Neda Nasiriani <[email protected]<mailto:[email protected]>> Sent: Wednesday, March 30, 2016 1:40 PM To: kusumakar bodha Cc: [email protected]<mailto:[email protected]> Subject: Re: Error media-streaming benchmark Hello kusumakar, I have the same problem and it happens on machines with more than 4 cores. If you try it on a machine with 4 or less cores this does not happen. It could be related to the cpu mask which is creating this problem for machines with >4 cores. I hope it get resolved. Please let me know if you have any findings. Thanks On Wednesday, March 30, 2016, kusumakar bodha <[email protected]<mailto:[email protected]>> wrote: Hi all, I'm trying to run the cloudsuite media-streaming benchmark and encountered the following error after I started the client. Total clients = 4 Minimum number of sessions = 25 Maximum number of sessions = 500 Launching 4 clients on localhost Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log --num-sessions=25 --rate=2 Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log --num-sessions=25 --rate=2 Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log --num-sessions=25 --rate=2 Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log --num-sessions=25 --rate=2 sizeof(fd_set) = 128 sizeof(fd_set) = 128 sizeof(fd_set) = 128 sizeof(fd_set) = 128 httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= Run on hostname: f2ef7e2c3e74 Run on hostname: f2ef7e2c3e74 httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=2 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= Run on hostname: f2ef7e2c3e74 Run at: Wed Mar 30 05:02:02 2016 Run on hostname: f2ef7e2c3e74 Run at: Wed Mar 30 05:02:02 2016 httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) Run at: Wed Mar 30 05:02:02 2016 httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) Run at: Wed Mar 30 05:02:02 2016 httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) grep: /output/*: No such file or directory grep: /output/*: No such file or directory peak_hunter/launch_hunt_bin.sh: line 56: 0*100/0: division by 0 (error token is "0") Benchmark succeeded for 25 sessions Launching 4 clients on localhost Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result1.log --num-sessions=500 --rate=50 Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result2.log --num-sessions=500 --rate=50 Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result3.log --num-sessions=500 --rate=50 Running command httperf --hog --server streaming_server --videosesslog=[/videos/logs/cl-240p-10-10000-10-5-00.log,/videos/logs/cl-360p-10-4000-10-5-00.log,/videos/logs/cl-480p-10-1600-10-5-00.log,/videos/logs/cl-720p-10-800-10-5-00.log],[0.1,0.3,0.4,0.2],[localhost,localhost,localhost,localhost] --epoll --recv-buffer=524288 --port 80 --output-log=/output/result4.log --num-sessions=500 --rate=50 sizeof(fd_set) = 128 httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= Run on hostname: f2ef7e2c3e74 sizeof(fd_set) = 128 Run at: Wed Mar 30 05:02:02 2016 httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= Run on hostname: f2ef7e2c3e74 Run at: Wed Mar 30 05:02:02 2016 httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) sizeof(fd_set) = 128 httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= Run on hostname: f2ef7e2c3e74 Run at: Wed Mar 30 05:02:03 2016 httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) sizeof(fd_set) = 128 httperf --hog --client=0/1 --server=streaming_server --port=80 --uri=/ --rate=50 --send-buffer=4096 --recv-buffer=524288 --epoll --call-stats-file= Run on hostname: f2ef7e2c3e74 Run at: Wed Mar 30 05:02:03 2016 httperf: sched_getaffinity failed, rc=-1 errno=22 (Invalid argument) grep: /output/*: No such file or directory grep: /output/*: No such file or directory peak_hunter/launch_hunt_bin.sh: line 56: 0*100/0: division by 0 (error token is "0") Benchmark succeeded for 500 sessions Maximum limit for number of sessions too low. grep: /output/*: No such file or directory (standard_in) 2: syntax error grep: /output/*: No such file or directory (standard_in) 2: syntax error grep: /output/*: No such file or directory (standard_in) 2: syntax error grep: /output/*: No such file or directory (standard_in) 2: syntax error grep: /output/*: No such file or directory (standard_in) 2: syntax error Requests: Replies: (standard_in) 1: syntax error Reply rate: (standard_in) 1: syntax error Reply time: Net I/O: What is the problem? The client is exiting immediately I'm using Docker version 1.9.0, build 76d6bc9 and kernel: 4.4.0-14-generic on ubuntu 14.04 Thank you Kusumakar -- Best Regards, Neda
