Re: [Puppet Users] Beaker - what’s your perspective?

[Trevor Vaughan]

Bram and Konrad,

I'm personally interested in your perspective on multi-node testing.

In my experience with our stack of around 100 modules, I've found that
single node testing is marginally better than spec testing in that it tells
you if a service actually starts. However, my project has found that
multi-node testing is the only way to know if most services (outside of
node-only services) actually function properly.

For instance, we found that perfectly valid rsyslog configurations would
cause the daemon to start and appear to be functioning properly but that
remote queueing, forwarding to multiple hosts, etc... was horribly broken.
This was only possible to troubleshoot with multi-node environments via
beaker (without writing something like beaker ourselves from scratch).

I understand that we're the minority but I think that it's a gap that has
significant real-world implications across the board.

Note, we do use InSpec for compliance testing and have actually integrated
it with Beaker in the simp-beaker-helpers gem.

Thanks,

Trevor

On Tue, Apr 21, 2020 at 2:51 PM Konrad Scherer 
wrote:

> On 4/21/20 3:33 AM, Bram Vogelaar wrote:
> > I too find it excruciating to use, I have described it in the past as
> having a rage inducing cli, and the lack of
> > documentation makes it hard to use. Which has driven me to use
> test-kitchen for doing puppet acceptance testing. (with
> > the added benefit of being able to use inspec profiles).
> > In my world view the uptake has been very small outside the vox
> pupuli and puppet supported modules because of this.
> > After Litmus was released i have never considered Beaker again for any
> real testing needs, since test-kitchen is very
> > very single node only by design.
>
> My experience was similar. I tried to figure out beaker but was unable to
> make progress due to the lack of documentation
> and examples (almost two years ago now). With test-kitchen and
> kitchen-puppet I was able to do my testing and it is my
> preferred runtime testing framework. I also found the integration with
> InSpec very useful.
>
> --
> Konrad Scherer, MTS, Linux Products Group, Wind River
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/e3571a4a-036c-5cea-7ae7-c92e8c3d08eb%40windriver.com
> .
>


-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CANs%2BFoWjv88hxUsxbudn0ZWVHRWmayuuGzy3OyMuBhd2PkBA1A%40mail.gmail.com.

Re: [Puppet Users] Beaker - what’s your perspective?

[Konrad Scherer]

On 4/21/20 3:33 AM, Bram Vogelaar wrote:
I too find it excruciating to use, I have described it in the past as having a rage inducing cli, and the lack of 
documentation makes it hard to use. Which has driven me to use test-kitchen for doing puppet acceptance testing. (with 
the added benefit of being able to use inspec profiles).
In my world view the uptake has been very small outside the vox pupuli and puppet supported modules because of this. 
After Litmus was released i have never considered Beaker again for any real testing needs, since test-kitchen is very 
very single node only by design.


My experience was similar. I tried to figure out beaker but was unable to make progress due to the lack of documentation 
and examples (almost two years ago now). With test-kitchen and kitchen-puppet I was able to do my testing and it is my 
preferred runtime testing framework. I also found the integration with InSpec very useful.


--
Konrad Scherer, MTS, Linux Products Group, Wind River

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/e3571a4a-036c-5cea-7ae7-c92e8c3d08eb%40windriver.com.

Re: [Puppet Users] Beaker - what’s your perspective?

[Bram Vogelaar]

I too find it excruciating to use, I have described it in the past as
having a rage inducing cli, and the lack of documentation makes it hard to
use. Which has driven me to use test-kitchen for doing puppet acceptance
testing. (with the added benefit of being able to use inspec profiles).
In my world view the uptake has been very small outside the vox pupuli and
puppet supported modules because of this. After Litmus was released i have
never considered Beaker again for any real testing needs, since
test-kitchen is very very single node only by design.





On Mon, 20 Apr 2020 at 20:41, 'Dan White' via Puppet Users <
puppet-users@googlegroups.com> wrote:

> Well put, Trevor.
> I have never used it because I have found it impossible to set up from
> scratch.
>
> "Sometimes I think the surest sign that intelligent life exists elsewhere
> in the universe is that none of it has tried to contact us."
>
> Bill Waterson (Calvin & Hobbes)
>
> On Apr 20, 2020, at 10:46 AM, Trevor Vaughan 
> wrote:
>
> 
> So, I chimed in over in Slack but wanted to go ahead and respond here with
> a summary of what we've been talking about there so that it'll be preserved
> for the future and searchable.
>
> This is a summary of multiple views and anyone participating in that
> discussion should feel free to correct my biased opinions in here (I like
> Beaker). I have a presentation that I did on exactly what I use Beaker for
> and why I like it from last year's conference at
> https://www.youtube.com/watch?v=4iBEIMQkBCk. The associated repository
> can be found at
> https://github.com/trevor-vaughan/puppetize_2019_multi_node_beaker for
> those that want a full working example.
>
> Who Uses it (module count is just modules, not necessarily modules with
> tests though the vast majority of the SIMP modules have tests):
>
>- Voxpupuli
>   - 127 Forge Modules
>   - The System Integrity Management Project
>   - 106 Forge Modules
>- A handful of other community folks
>   - I'd love to see a full analysis of all forge modules and what
>   type of testing they use but I don't have time to dig into that right 
> now
>   (Gene?)
>
> The Pros:
>
>- Beaker generally works as it is for both single node and multi-node
>(my main use case) testing.
>   - See the video as to why multi-node testing is important
>   - It preserves the rspec syntax that makes the output of the tests
>easy to understand for non-technical folks as well as easy to trace for
>technical folks.
>- It has the ability to be extended relatively easily in Ruby
>- It works with most major cloud providers (and Vagrant)
>- It hasn't really taken a lot of care and feeding recently to keep it
>chugging along
>
> The Cons:
>
>- It's not well documented (at all)
>   - When the project was modularized a couple of years ago, the
>   documentation was thrown to the four winds with each of the modules and 
> the
>   care and feeding of the docs pretty much dried up.
>- The DSL is inconsistent. Some methods are 'on(host)' others are
>'host.thing()' which is pretty darn confusing
>- Since it hasn't had a ton of internal care and feeding, it hasn't
>kept up with all of the things that Bolt can do. On the other hand, it also
>seems to have solved some issues that are currently being faced by the next
>generation of proposed testing tech.
>
>
> Thanks,
>
> Trevor
>
> On Fri, Apr 17, 2020 at 7:40 AM Gene Liverman 
> wrote:
>
>> Hi friends! I’m trying to better understand the community perspective on
>> Beaker and its supplemental gems. I’m particularly interested in hearing
>> your thoughts on the state of its maintenance and what, if anything, you’d
>> like to see change in that regard. I’m looking for both positive and
>> negative opinions and impressions.
>> --
>>
>>
>>
>> Gene Liverman
>> Sr. Site Reliability Engineer
>> gene.liver...@puppet.com
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to puppet-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/CA%2BmGaMcuF0OrPjjkSqJdb%2Bd_Nq9vSN0c3k8wP4L1v2SSZ-7Htw%40mail.gmail.com
>> 
>> .
>>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699 x788
>
> -- This account not approved for unencrypted proprietary information --
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> 

Re: [Puppet Users] Beaker - what’s your perspective?

['Dan White' via Puppet Users]

Well put, Trevor. 
I have never used it because I have found it impossible to set up from scratch. 

"Sometimes I think the surest sign that intelligent life exists elsewhere in 
the universe is that none of it has tried to contact us."
Bill Waterson (Calvin & Hobbes)

> On Apr 20, 2020, at 10:46 AM, Trevor Vaughan  wrote:
> 
> 
> So, I chimed in over in Slack but wanted to go ahead and respond here with a 
> summary of what we've been talking about there so that it'll be preserved for 
> the future and searchable.
> 
> This is a summary of multiple views and anyone participating in that 
> discussion should feel free to correct my biased opinions in here (I like 
> Beaker). I have a presentation that I did on exactly what I use Beaker for 
> and why I like it from last year's conference at 
> https://www.youtube.com/watch?v=4iBEIMQkBCk. The associated repository can be 
> found at https://github.com/trevor-vaughan/puppetize_2019_multi_node_beaker 
> for those that want a full working example.
> 
> Who Uses it (module count is just modules, not necessarily modules with tests 
> though the vast majority of the SIMP modules have tests):
> Voxpupuli
> 127 Forge Modules
> The System Integrity Management Project
> 106 Forge Modules
> A handful of other community folks
> I'd love to see a full analysis of all forge modules and what type of testing 
> they use but I don't have time to dig into that right now (Gene?)
> The Pros:
> Beaker generally works as it is for both single node and multi-node (my main 
> use case) testing.
> See the video as to why multi-node testing is important
> It preserves the rspec syntax that makes the output of the tests easy to 
> understand for non-technical folks as well as easy to trace for technical 
> folks.
> It has the ability to be extended relatively easily in Ruby
> It works with most major cloud providers (and Vagrant)
> It hasn't really taken a lot of care and feeding recently to keep it chugging 
> along
> The Cons:
> It's not well documented (at all)
> When the project was modularized a couple of years ago, the documentation was 
> thrown to the four winds with each of the modules and the care and feeding of 
> the docs pretty much dried up.
> The DSL is inconsistent. Some methods are 'on(host)' others are 
> 'host.thing()' which is pretty darn confusing
> Since it hasn't had a ton of internal care and feeding, it hasn't kept up 
> with all of the things that Bolt can do. On the other hand, it also seems to 
> have solved some issues that are currently being faced by the next generation 
> of proposed testing tech.
> 
> Thanks,
> 
> Trevor
> 
>> On Fri, Apr 17, 2020 at 7:40 AM Gene Liverman  
>> wrote:
>> Hi friends! I’m trying to better understand the community perspective on 
>> Beaker and its supplemental gems. I’m particularly interested in hearing 
>> your thoughts on the state of its maintenance and what, if anything, you’d 
>> like to see change in that regard. I’m looking for both positive and 
>> negative opinions and impressions. 
>> -- 
>> 
>> 
>> 
>> Gene Liverman
>> Sr. Site Reliability Engineer
>> gene.liver...@puppet.com
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to puppet-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/puppet-users/CA%2BmGaMcuF0OrPjjkSqJdb%2Bd_Nq9vSN0c3k8wP4L1v2SSZ-7Htw%40mail.gmail.com.
> 
> 
> -- 
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699 x788
> 
> -- This account not approved for unencrypted proprietary information --
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/puppet-users/CANs%2BFoXCtGHEbo8ENO9Cd_RcAJBjKKHqg67OTpUDvUMoF69kaA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/62A0053C-FE8E-44F4-9883-AB9918DE418B%40icloud.com.

Re: [Puppet Users] Beaker - what’s your perspective?

[Trevor Vaughan]

So, I chimed in over in Slack but wanted to go ahead and respond here with
a summary of what we've been talking about there so that it'll be preserved
for the future and searchable.

This is a summary of multiple views and anyone participating in that
discussion should feel free to correct my biased opinions in here (I like
Beaker). I have a presentation that I did on exactly what I use Beaker for
and why I like it from last year's conference at
https://www.youtube.com/watch?v=4iBEIMQkBCk. The associated repository can
be found at
https://github.com/trevor-vaughan/puppetize_2019_multi_node_beaker for
those that want a full working example.

Who Uses it (module count is just modules, not necessarily modules with
tests though the vast majority of the SIMP modules have tests):

   - Voxpupuli
  - 127 Forge Modules
  - The System Integrity Management Project
  - 106 Forge Modules
   - A handful of other community folks
  - I'd love to see a full analysis of all forge modules and what type
  of testing they use but I don't have time to dig into that right
now (Gene?)

The Pros:

   - Beaker generally works as it is for both single node and multi-node
   (my main use case) testing.
  - See the video as to why multi-node testing is important
  - It preserves the rspec syntax that makes the output of the tests
   easy to understand for non-technical folks as well as easy to trace for
   technical folks.
   - It has the ability to be extended relatively easily in Ruby
   - It works with most major cloud providers (and Vagrant)
   - It hasn't really taken a lot of care and feeding recently to keep it
   chugging along

The Cons:

   - It's not well documented (at all)
  - When the project was modularized a couple of years ago, the
  documentation was thrown to the four winds with each of the
modules and the
  care and feeding of the docs pretty much dried up.
   - The DSL is inconsistent. Some methods are 'on(host)' others are
   'host.thing()' which is pretty darn confusing
   - Since it hasn't had a ton of internal care and feeding, it hasn't kept
   up with all of the things that Bolt can do. On the other hand, it also
   seems to have solved some issues that are currently being faced by the next
   generation of proposed testing tech.


Thanks,

Trevor

On Fri, Apr 17, 2020 at 7:40 AM Gene Liverman 
wrote:

> Hi friends! I’m trying to better understand the community perspective on
> Beaker and its supplemental gems. I’m particularly interested in hearing
> your thoughts on the state of its maintenance and what, if anything, you’d
> like to see change in that regard. I’m looking for both positive and
> negative opinions and impressions.
> --
>
>
>
> Gene Liverman
> Sr. Site Reliability Engineer
> gene.liver...@puppet.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CA%2BmGaMcuF0OrPjjkSqJdb%2Bd_Nq9vSN0c3k8wP4L1v2SSZ-7Htw%40mail.gmail.com
> 
> .
>


-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CANs%2BFoXCtGHEbo8ENO9Cd_RcAJBjKKHqg67OTpUDvUMoF69kaA%40mail.gmail.com.

[Puppet Users] Beaker - what’s your perspective?

[Gene Liverman]

Hi friends! I’m trying to better understand the community perspective on
Beaker and its supplemental gems. I’m particularly interested in hearing
your thoughts on the state of its maintenance and what, if anything, you’d
like to see change in that regard. I’m looking for both positive and
negative opinions and impressions.
-- 



Gene Liverman
Sr. Site Reliability Engineer
gene.liver...@puppet.com

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CA%2BmGaMcuF0OrPjjkSqJdb%2Bd_Nq9vSN0c3k8wP4L1v2SSZ-7Htw%40mail.gmail.com.