Review Request 27111: Block IO Isolator: usage metrics

2014-10-23 Thread Joris Van Remoortere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/
---

Review request for mesos, Benjamin Hindman and Ian Downes.


Bugs: MESOS-1977
https://issues.apache.org/jira/browse/MESOS-1977


Repository: mesos-git


Description
---

Implement a block io isolator that just publishes read / write, bytes / 
operations per second.
A split of r25922 as suggested by Ian.


Diffs
-

  include/mesos/mesos.proto 6b93e90 
  src/Makefile.am 2617f77 
  src/linux/cgroups.hpp abf31df 
  src/linux/cgroups.cpp 62df4b7 
  src/linux/fs.hpp ac8b5f4 
  src/linux/fs.cpp b01d14c 
  src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
  src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
  src/slave/containerizer/mesos/containerizer.cpp 9d08329 
  src/tests/isolator_tests.cpp 52b38a3 

Diff: https://reviews.apache.org/r/27111/diff/


Testing
---

make check
sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"


Thanks,

Joris Van Remoortere

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-23 Thread Mesos ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/#review58109
---


Bad patch!

Reviews applied: [27111]

Failed command: ./support/apply-review.sh -n -r 27111

Error:
 --2014-10-23 21:20:04--  https://reviews.apache.org/r/27111/diff/raw/
Resolving reviews.apache.org (reviews.apache.org)... 140.211.11.74
Connecting to reviews.apache.org (reviews.apache.org)|140.211.11.74|:443... 
connected.
HTTP request sent, awaiting response... 200 OK
Length: 26290 (26K) [text/x-patch]
Saving to: '27111.patch'

 0K .. .. .   100%  780K=0.03s

2014-10-23 21:20:04 (780 KB/s) - '27111.patch' saved [26290/26290]

Successfully applied: Block IO Isolator: usage metrics

Implement a block io isolator that just publishes read / write, bytes / 
operations per second.
A split of r25922 as suggested by Ian.


Review: https://reviews.apache.org/r/27111
Checking 8 files using filter 
--filter=-,+build/class,+build/deprecated,+build/endif_comment,+readability/todo,+readability/namespace,+runtime/vlog,+whitespace/blank_line,+whitespace/comma,+whitespace/end_of_line,+whitespace/ending_newline,+whitespace/forcolon,+whitespace/indent,+whitespace/line_length,+whitespace/operators,+whitespace/semicolon,+whitespace/tab,+whitespace/todo
src/tests/isolator_tests.cpp:910:  Could not find a newline character at the 
end of the file.  [whitespace/ending_newline] [5]
src/slave/containerizer/isolators/cgroups/blkio.cpp:217:  Redundant blank line 
at the start of a code block should be deleted.  [whitespace/blank_line] [2]
Total errors found: 2
Failed to commit patch

- Mesos ReviewBot


On Oct. 23, 2014, 9:01 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27111/
> ---
> 
> (Updated Oct. 23, 2014, 9:01 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Ian Downes.
> 
> 
> Bugs: MESOS-1977
> https://issues.apache.org/jira/browse/MESOS-1977
> 
> 
> Repository: mesos-git
> 
> 
> Description
> ---
> 
> Implement a block io isolator that just publishes read / write, bytes / 
> operations per second.
> A split of r25922 as suggested by Ian.
> 
> 
> Diffs
> -
> 
>   include/mesos/mesos.proto 6b93e90 
>   src/Makefile.am 2617f77 
>   src/linux/cgroups.hpp abf31df 
>   src/linux/cgroups.cpp 62df4b7 
>   src/linux/fs.hpp ac8b5f4 
>   src/linux/fs.cpp b01d14c 
>   src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
>   src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
>   src/slave/containerizer/mesos/containerizer.cpp 9d08329 
>   src/tests/isolator_tests.cpp 52b38a3 
> 
> Diff: https://reviews.apache.org/r/27111/diff/
> 
> 
> Testing
> ---
> 
> make check
> sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-23 Thread Joris Van Remoortere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/
---

(Updated Oct. 23, 2014, 9:58 p.m.)


Review request for mesos, Benjamin Hindman and Ian Downes.


Changes
---

fix whitespace issues.


Bugs: MESOS-1977
https://issues.apache.org/jira/browse/MESOS-1977


Repository: mesos-git


Description
---

Implement a block io isolator that just publishes read / write, bytes / 
operations per second.
A split of r25922 as suggested by Ian.


Diffs (updated)
-

  include/mesos/mesos.proto 6b93e90 
  src/Makefile.am 2617f77 
  src/linux/cgroups.hpp abf31df 
  src/linux/cgroups.cpp 62df4b7 
  src/linux/fs.hpp ac8b5f4 
  src/linux/fs.cpp b01d14c 
  src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
  src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
  src/slave/containerizer/mesos/containerizer.cpp 9d08329 
  src/tests/isolator_tests.cpp 52b38a3 

Diff: https://reviews.apache.org/r/27111/diff/


Testing
---

make check
sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"


Thanks,

Joris Van Remoortere

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-23 Thread Ben Mahler 


> On Oct. 23, 2014, 9:20 p.m., Mesos ReviewBot wrote:
> > Bad patch!
> > 
> > Reviews applied: [27111]
> > 
> > Failed command: ./support/apply-review.sh -n -r 27111
> > 
> > Error:
> >  --2014-10-23 21:20:04--  https://reviews.apache.org/r/27111/diff/raw/
> > Resolving reviews.apache.org (reviews.apache.org)... 140.211.11.74
> > Connecting to reviews.apache.org (reviews.apache.org)|140.211.11.74|:443... 
> > connected.
> > HTTP request sent, awaiting response... 200 OK
> > Length: 26290 (26K) [text/x-patch]
> > Saving to: '27111.patch'
> > 
> >  0K .. .. .   100%  
> > 780K=0.03s
> > 
> > 2014-10-23 21:20:04 (780 KB/s) - '27111.patch' saved [26290/26290]
> > 
> > Successfully applied: Block IO Isolator: usage metrics
> > 
> > Implement a block io isolator that just publishes read / write, bytes / 
> > operations per second.
> > A split of r25922 as suggested by Ian.
> > 
> > 
> > Review: https://reviews.apache.org/r/27111
> > Checking 8 files using filter 
> > --filter=-,+build/class,+build/deprecated,+build/endif_comment,+readability/todo,+readability/namespace,+runtime/vlog,+whitespace/blank_line,+whitespace/comma,+whitespace/end_of_line,+whitespace/ending_newline,+whitespace/forcolon,+whitespace/indent,+whitespace/line_length,+whitespace/operators,+whitespace/semicolon,+whitespace/tab,+whitespace/todo
> > src/tests/isolator_tests.cpp:910:  Could not find a newline character at 
> > the end of the file.  [whitespace/ending_newline] [5]
> > src/slave/containerizer/isolators/cgroups/blkio.cpp:217:  Redundant blank 
> > line at the start of a code block should be deleted.  
> > [whitespace/blank_line] [2]
> > Total errors found: 2
> > Failed to commit patch

Are you using the commit hook in support/ ?


- Ben


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/#review58109
---


On Oct. 23, 2014, 9:58 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27111/
> ---
> 
> (Updated Oct. 23, 2014, 9:58 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Ian Downes.
> 
> 
> Bugs: MESOS-1977
> https://issues.apache.org/jira/browse/MESOS-1977
> 
> 
> Repository: mesos-git
> 
> 
> Description
> ---
> 
> Implement a block io isolator that just publishes read / write, bytes / 
> operations per second.
> A split of r25922 as suggested by Ian.
> 
> 
> Diffs
> -
> 
>   include/mesos/mesos.proto 6b93e90 
>   src/Makefile.am 2617f77 
>   src/linux/cgroups.hpp abf31df 
>   src/linux/cgroups.cpp 62df4b7 
>   src/linux/fs.hpp ac8b5f4 
>   src/linux/fs.cpp b01d14c 
>   src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
>   src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
>   src/slave/containerizer/mesos/containerizer.cpp 9d08329 
>   src/tests/isolator_tests.cpp 52b38a3 
> 
> Diff: https://reviews.apache.org/r/27111/diff/
> 
> 
> Testing
> ---
> 
> make check
> sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-23 Thread Mesos ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/#review58221
---


Patch looks great!

Reviews applied: [27111]

All tests passed.

- Mesos ReviewBot


On Oct. 23, 2014, 9:58 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27111/
> ---
> 
> (Updated Oct. 23, 2014, 9:58 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Ian Downes.
> 
> 
> Bugs: MESOS-1977
> https://issues.apache.org/jira/browse/MESOS-1977
> 
> 
> Repository: mesos-git
> 
> 
> Description
> ---
> 
> Implement a block io isolator that just publishes read / write, bytes / 
> operations per second.
> A split of r25922 as suggested by Ian.
> 
> 
> Diffs
> -
> 
>   include/mesos/mesos.proto 6b93e90 
>   src/Makefile.am 2617f77 
>   src/linux/cgroups.hpp abf31df 
>   src/linux/cgroups.cpp 62df4b7 
>   src/linux/fs.hpp ac8b5f4 
>   src/linux/fs.cpp b01d14c 
>   src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
>   src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
>   src/slave/containerizer/mesos/containerizer.cpp 9d08329 
>   src/tests/isolator_tests.cpp 52b38a3 
> 
> Diff: https://reviews.apache.org/r/27111/diff/
> 
> 
> Testing
> ---
> 
> make check
> sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-24 Thread Ian Downes 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/#review58418
---


Close to ready. Major points are cleaning up the cgroups statistics functions 
and making the test child safe.


src/linux/cgroups.hpp


s/The a pair/The pair/



src/linux/cgroups.cpp


We accept ">>" now.



src/linux/cgroups.cpp


I think it's error-prone to have the implicit  pairing and 
specifying the file.

What about hiding this function in an internal namespace and exposing the 
various statistics explicitly under a descriptive namespace, e.g., perhaps 
something like this:

Bytes blkio::statistics::bytes::{read,write}()

uint64_t blkio::statistics::iops::{read,write}()

Yes, converting to Bytes to return is unnecessary in your usage but it 
avoids any ambiguity for the user of this library function.



src/linux/fs.cpp


Any tests for this?



src/linux/fs.cpp


move this to after the file.fail() check?



src/slave/containerizer/isolators/cgroups/blkio.cpp


kill newline



src/slave/containerizer/isolators/cgroups/blkio.cpp


What's the difference between blkio.throttle.io_service_bytes and  
blkio.io_service_bytes?



src/tests/isolator_tests.cpp


s/alligned/aligned/



src/tests/isolator_tests.cpp


posix_memalign is definitely not async-signal-safe



src/tests/isolator_tests.cpp


s/ret/write/? or count or ...
something other than 'ret', please :-)

pwrite() is not listed as async-signal-safe: 
http://man7.org/linux/man-pages/man7/signal.7.html



src/tests/isolator_tests.cpp


ditto



src/tests/isolator_tests.cpp


This "setup" function must be async-signal-safe as it's called between the 
fork and exec, and we're forking from a threaded parent.

http://man7.org/linux/man-pages/man7/signal.7.html

Can you verify that everything in there is safe (and comment that it is) or 
put this code into a separate helper binary?


- Ian Downes


On Oct. 23, 2014, 2:58 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27111/
> ---
> 
> (Updated Oct. 23, 2014, 2:58 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Ian Downes.
> 
> 
> Bugs: MESOS-1977
> https://issues.apache.org/jira/browse/MESOS-1977
> 
> 
> Repository: mesos-git
> 
> 
> Description
> ---
> 
> Implement a block io isolator that just publishes read / write, bytes / 
> operations per second.
> A split of r25922 as suggested by Ian.
> 
> 
> Diffs
> -
> 
>   include/mesos/mesos.proto 6b93e90 
>   src/Makefile.am 2617f77 
>   src/linux/cgroups.hpp abf31df 
>   src/linux/cgroups.cpp 62df4b7 
>   src/linux/fs.hpp ac8b5f4 
>   src/linux/fs.cpp b01d14c 
>   src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
>   src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
>   src/slave/containerizer/mesos/containerizer.cpp 9d08329 
>   src/tests/isolator_tests.cpp 52b38a3 
> 
> Diff: https://reviews.apache.org/r/27111/diff/
> 
> 
> Testing
> ---
> 
> make check
> sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-25 Thread Joris Van Remoortere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/
---

(Updated Oct. 25, 2014, 5:48 p.m.)


Review request for mesos, Benjamin Hindman and Ian Downes.


Changes
---

Re-organize blkio stats.
Make blkio unit test setup async-signal-safe.


Bugs: MESOS-1977
https://issues.apache.org/jira/browse/MESOS-1977


Repository: mesos-git


Description
---

Implement a block io isolator that just publishes read / write, bytes / 
operations per second.
A split of r25922 as suggested by Ian.


Diffs (updated)
-

  include/mesos/mesos.proto 6b93e90 
  src/Makefile.am 2617f77 
  src/linux/cgroups.hpp abf31df 
  src/linux/cgroups.cpp 62df4b7 
  src/linux/fs.hpp ac8b5f4 
  src/linux/fs.cpp b01d14c 
  src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
  src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
  src/slave/containerizer/mesos/containerizer.cpp 9d08329 
  src/tests/isolator_tests.cpp 52b38a3 

Diff: https://reviews.apache.org/r/27111/diff/


Testing
---

make check
sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"


Thanks,

Joris Van Remoortere

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-25 Thread Joris Van Remoortere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/
---

(Updated Oct. 25, 2014, 5:53 p.m.)


Review request for mesos, Benjamin Hindman and Ian Downes.


Changes
---

Add async-signal-safe comments.
Fix cosmetics.


Bugs: MESOS-1977
https://issues.apache.org/jira/browse/MESOS-1977


Repository: mesos-git


Description
---

Implement a block io isolator that just publishes read / write, bytes / 
operations per second.
A split of r25922 as suggested by Ian.


Diffs (updated)
-

  include/mesos/mesos.proto 6b93e90 
  src/Makefile.am 2617f77 
  src/linux/cgroups.hpp abf31df 
  src/linux/cgroups.cpp 62df4b7 
  src/linux/fs.hpp ac8b5f4 
  src/linux/fs.cpp b01d14c 
  src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
  src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
  src/slave/containerizer/mesos/containerizer.cpp 9d08329 
  src/tests/isolator_tests.cpp 52b38a3 

Diff: https://reviews.apache.org/r/27111/diff/


Testing
---

make check
sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"


Thanks,

Joris Van Remoortere

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-25 Thread Joris Van Remoortere 


> On Oct. 25, 2014, 1:02 a.m., Ian Downes wrote:
> > src/slave/containerizer/isolators/cgroups/blkio.cpp, line 252
> > 
> >
> > What's the difference between blkio.throttle.io_service_bytes and  
> > blkio.io_service_bytes?

>From my experimentation: blkio.io_service_bytes does not actually report the 
>bytes read/written, whereas blkio.throttle.io_service_bytes does.


- Joris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/#review58418
---


On Oct. 25, 2014, 5:53 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27111/
> ---
> 
> (Updated Oct. 25, 2014, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Ian Downes.
> 
> 
> Bugs: MESOS-1977
> https://issues.apache.org/jira/browse/MESOS-1977
> 
> 
> Repository: mesos-git
> 
> 
> Description
> ---
> 
> Implement a block io isolator that just publishes read / write, bytes / 
> operations per second.
> A split of r25922 as suggested by Ian.
> 
> 
> Diffs
> -
> 
>   include/mesos/mesos.proto 6b93e90 
>   src/Makefile.am 2617f77 
>   src/linux/cgroups.hpp abf31df 
>   src/linux/cgroups.cpp 62df4b7 
>   src/linux/fs.hpp ac8b5f4 
>   src/linux/fs.cpp b01d14c 
>   src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
>   src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
>   src/slave/containerizer/mesos/containerizer.cpp 9d08329 
>   src/tests/isolator_tests.cpp 52b38a3 
> 
> Diff: https://reviews.apache.org/r/27111/diff/
> 
> 
> Testing
> ---
> 
> make check
> sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>

Re: Review Request 27111: Block IO Isolator: usage metrics

2014-10-25 Thread Mesos ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27111/#review58524
---


Patch looks great!

Reviews applied: [27111]

All tests passed.

- Mesos ReviewBot


On Oct. 25, 2014, 5:53 p.m., Joris Van Remoortere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27111/
> ---
> 
> (Updated Oct. 25, 2014, 5:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Ian Downes.
> 
> 
> Bugs: MESOS-1977
> https://issues.apache.org/jira/browse/MESOS-1977
> 
> 
> Repository: mesos-git
> 
> 
> Description
> ---
> 
> Implement a block io isolator that just publishes read / write, bytes / 
> operations per second.
> A split of r25922 as suggested by Ian.
> 
> 
> Diffs
> -
> 
>   include/mesos/mesos.proto 6b93e90 
>   src/Makefile.am 2617f77 
>   src/linux/cgroups.hpp abf31df 
>   src/linux/cgroups.cpp 62df4b7 
>   src/linux/fs.hpp ac8b5f4 
>   src/linux/fs.cpp b01d14c 
>   src/slave/containerizer/isolators/cgroups/blkio.hpp PRE-CREATION 
>   src/slave/containerizer/isolators/cgroups/blkio.cpp PRE-CREATION 
>   src/slave/containerizer/mesos/containerizer.cpp 9d08329 
>   src/tests/isolator_tests.cpp 52b38a3 
> 
> Diff: https://reviews.apache.org/r/27111/diff/
> 
> 
> Testing
> ---
> 
> make check
> sudo ./mesos-tests --gtest_filter="BlkIOIsolatorTest*"
> 
> 
> Thanks,
> 
> Joris Van Remoortere
> 
>