Working at Puppet

Posted on by
1

I work for Puppet Labs, as is likely obvious at this point, and I talk a lot about how great it is as a company. This may read like corporate shilling, but it’s not – I don’t get asked to or paid for writing this stuff…it’s a matter of spreading the word about how technology should be done, and I believe we do it right.

I had a chance recently to contribute to a magazine survey about great places to work, and I submitted a really overlong entry (really? me? how out of character…) But I meant everything I said, and I thought I’d share the full thing here:

It’s really difficult to distill everything great about Puppet down into anything brief, so I’ll go for something emblematic. When I asked Luke Kanies, our CEO, what single book he’d most recommend I read to understand his philosophy, his immediate answer without a moment’s pause was “The Pragmatic Programmer.” For those in the business who know about this book, it’s a how-to manual for Doing It Right – being a responsible contributor to a company who is realistic about his capabilities and maximizes both his/her personal and professional potential.

Puppet Labs is this methodology in practice. Is it always perfect in achieving this? Not directly, but internal criticism is not just well-received, it’s encouraged and acted upon – and the ability to incrementally improve your own ability to reach an ideal is exactly what this philosophy is all about.

The fact that, as someone in a Professional Services area, I was in a position to directly and casually ask our CEO for a book recommendation is of course another reason Puppet is great. The discussion revolved around me asking Luke how I could break into development, something nearly impossible in many companies, and yet something Luke actively encouraged me to do. In fact, not only did he recommend that one book, he rattled off a few more, then did a blog post with links to their locations and sent me an e-mail the next morning offering to recommend more when I’d finished those.

The next morning I worked up a quick prototype and ran it to my boss, James Turnbull, to ensure that it was acceptable for me to spend my time working on stuff outside my group’s spec. Not only did he say it was alright, he actively encouraged me to document and share the work, and apply for an internal company grant to help get more resources on it. I received the same warm welcome from the development team, who made it absolutely clear that they wanted to make it as easy as possible for me to begin contributing to more areas of the company.

It’s rare that you get a chance to work with some of the brightest people in the industry. It’s rare that you have a supervisor who will let you work on projects outside of the scope of your job. It’s rare that you have a separate team willing to accept work from an “outsider.” It’s rare to have a CEO who is approachable enough for me to be comfortable having the discussion I did, and is so knowledgable that those recommendations are almost immediately career-changing. To have them all in one place? That’s Puppet.

Building Colloquy from Source on Lion with Xcode 4.2

Posted on by
1

(UPDATE 4/2012) – A new version of Colloquy is out with Growl support working properly.  The below should no longer be necessary.

(Updated to fix formatting issue)

I’ve got the misfortune of being addicted to the use of Colloquy on my Mac for IRC.  Life would be easy if I could just be addicted to irssi, but for whatever reason, despite the myriad color schemes and output formats available, my brain just doesn’t parse IRC output from most IRC clients with the clarity it gets from Colloquy.  I’m sure I could tweak the heck out of irssi to make it work for me…but I digress.

For a very long time Colloquy’s had a problem with Growl support – if you’ve used it, you are probably familiar with the issue that Colloquy overrides Growl’s default appearances settings and chooses the most obnoxious, persistent option possible.  The fix for this used to be running Colloquy in 32-bit mode, not a huge problem once you figured it out.

But I’m one of the adventurous masochistic folk who have chosen to upgrade to Growl 1.3, mainly out of a desire to support Growl’s continuing development.  This has broken backwards compatibility with a ton of apps, and the Growl devs are working with many of these apps to get fixes in place.

Colloquy has a fix in its latest SVN commits, but building Colloquy under Lion 10.7.2 and Xcode 4.2 has proven to be an adventure.  The following blog post got me nearly all the way there:

A fix for the Colloquy timestamp bug in OS X Lion

But missed a couple of details that tripped me up.  So what follows are the instructions from the above blog, fleshed out with the few additional steps I needed.

  1. svn checkout http://source.colloquy.info/svn/trunk Colloquy
  2. Download F-Script from fscript.org.  The zip file contains a directory called “Fscript.framework” – you should find a directory of the same name under /Library/Frameworks.  Delete the existing Fscript.framework directory and replace it with the one from the archive.
  3. Open Colloquy (Old).xcodeproj
  4. At the top of the app, next to the run and stop buttons, you will see a drop-down allowing you to select your build target.  Select AGRegex (framework).  Then click Product -> Build.  You should see some warnings, but the build should succeed.
  5. Select the Colloquy (Application) target from the drop down at the top.
  6. For some reason, I needed to change the path in a number of the source files for one particular library.  There is probably a really easy, obvious namespace fix to correct this that would take me almost no googling at all.  But since there are only 4-5 references in the entire project, searching for the reference and replacing it with the absolute path was much easier.  Just look for
  7. #include <ChatCore

    and replace it with

    #include ChatCore
  8. Product -> Build.  If this is successful, select Product -> Build For -> Build For Running.
  9. Your Colloquy.app is in Colloquy/build/Development.  Just drag and drop into your Application folder like any other app.

There is no longer any need to run in 32-bit mode – 64 works fine.

This has been running and quite stable for me for a few hours…if it becomes less stable I’ll post back.

Chariot Seminar Cancelled

Posted on by
Reply

According to Ken Rimple at Chariot Solutions, their scheduled DevOps seminar has been cancelled.

This is really a shame, because it offered a chance to reach out to a group that has been underexposed to Puppet.  But we’ll regroup, and hopefully have a chance to present something soon in the future.

You can still find the second part of our podcast discussing Puppet and DevOps in general here.

Cutter IT Journal

Posted on by
Reply

An article I wrote back when I worked at Advance Internet has finally come out in this month’s Cutter IT Journal. The magazine normally costs a bunch, but you can get this month’s issue for free at the following link: http://bit.ly/p60uvP The magazine was guest-edited by Patrick Debois and features a ton of great content on real-world DevOps implementations. Worth a read for anybody in IT.

Chariot TechCast, 8/20/2011

Posted on by
Reply
Ken Rimple at Chariot Solutions has posted the first part of a two-part interview we did the other day on DevOps and Puppet: http://techcast.chariotsolutions.com/chariot-tech-cast-65-discussion-with-puppet-labs-eric-shamow-part-1# You should check out the TechCast in general, lots of coverage of current development technologies and practices, and Ken’s a sharp guy. Part two goes much more in-depth about Puppet, discussing some of the features coming this year and PuppetConf. Part two should be up in early September.

Chariot Devops Seminar

Posted on by
Reply

I’ll be speaking at the upcoming Chariot DevOps seminar on Puppet — I’ll be doing a technical demo — and then DevOps in general as part of a panel discussion. The event is October 18th at the Penn State Great Valley campus in Malvern, Pennsylvania. You can find out more here: Chariot Solutions DevOps Seminar On another note, if you haven’t signed up for PuppetConf, what are you waiting for? September 22nd and 23rd in Portland – it’s a general operations conference, so while Puppet will be featured, there will also be tracks on cloud technologies and DevOps. Portland’s a great town and the Puppet folks are amazing people – this is a chance to talk to and influence some really smart people. Go.

Mirroring RubyGems, and Ruby 9.2 on CentOS 5.5

Posted on by
Reply

This is one of those troubleshooting scenarios that starts out being about one thing (in this case, mirroring RubyForge) and ends up being about something entirely different (getting a new Ruby running in an old CentOS).  To wit: One of the most frustrating things about running new code in stable production environments is the assumption by many developers that all of your servers will have access to the outside world, and to their repositories. For general security, unless a host is an edge web server, we don’t allow outbound connections to untrusted subnets from our systems, let alone to repos on the outside world.  And even if we did allow it, I really don’t need my hosts phoning out to third-party repos all over the place and asking for data.  Those hosts are internal and should stay that way. To get around this we tend to mirror repositories locally with Cobbler, and point our hosts at these repos.  Generally that’s all well and good since we are mostly a CentOS shop and use RPM based repos.  But as part of our recent efforts to deploy cucumber-puppet, I decided to mirror RubyGems in order to allow a small number of test hosts to install via gem instead of RPM (another post later on the perils of mixing those two…).  The saga of getting this running was interesting and worth noting.  (I should note the considerable assistance of Sun’s BigAdmin pages, which I never thought would ever come in handy again, buthttp://www.sun.com/bigadmin/content/submitted/ruby_http.jsp — pretty good starting point). To begin, if you’re not able to route out but can get to a proxy from the box hosting your mirrored repos (how else are you going to get them?), you went the builder gem installed.  You can get this via “gem install builder,” or build/download an RPM, whichever suits your package management scheme.  WIth this done, create a file in /root/.gemmirrorrc containing:

---
- from: http://gems.rubyforge.org/
  to: /gemrepo/dev

Replace the to: field with the location of your target repo. Rubygems are nice enough to obey your proxy settings, so go ahead and execute:

export http_proxy=http://<yourproxyserver>

and with this done, simply run gem mirror To download the gems from RubyForge.  Be prepared for a very, very long wait — gem files are small, but there are around 40,000.  Go get coffee, or better yet dinner. With that done, the BigAdmin page asks you to generate an index, a process similar to the createrepo command.  This sounds good in practice — just run

gem generate_index -d /gemrepo/dev

Again substituting your directory for /gemrepo/dev.  And here is where we ran into the fun, after just a few minutes.  I’ll be happy to post the mountain of stack trace fail if anyone is interested but the critical bits, from the beginning and end of the trace, are here:

...............................................................*** buffer overflow detected ***: /usr/bin/ruby terminated

======= Backtrace: =========
/lib64/libc.so.6(__chk_fail+0x2f)[0x33e12e803f]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(rb_syck_mktime+0x48e)[0x2b5b11b36a7e]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(yaml_org_handler+0x860)[0x2b5b11b37390]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_defaultresolver_node_import+0x39)[0x2b5b11b37599]
/usr/lib64/libruby.so.1.8[0x33e1a34a11]
/usr/lib64/libruby.so.1.8[0x33e1a34f28]
/usr/lib64/libruby.so.1.8[0x33e1a3552a]
/usr/lib64/libruby.so.1.8(rb_funcall+0x85)[0x33e1a357f5]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(rb_syck_load_handler+0x47)[0x2b5b11b36527]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_hdlr_add_node+0x39)[0x2b5b11b30d49]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(syckparse+0xb45)[0x2b5b11b305d5]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_parse+0x19)[0x2b5b11b38169]
/usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_parser_load+0xed)[0x2b5b11b3639d]

...

00400000-00401000 r-xp 00000000 fd:00 1291456                            /usr/bin/ruby

00600000-00602000 rw-p 00000000 fd:00 1291456                            /usr/bin/ruby
1a46b000-296d6000 rw-p 1a46b000 00:00 0                                  [heap]
33e0e00000-33e0e1c000 r-xp 00000000 fd:00 1998852                        /lib64/ld-2.5.so
33e101b000-33e101c000 r--p 0001b000 fd:00 1998852                        /lib64/ld-2.5.so
33e101c000-33e101d000 rw-p 0001c000 fd:00 1998852                        /lib64/ld-2.5.so
33e1200000-33e134e000 r-xp 00000000 fd:00 1998859                        /lib64/libc-2.5.so
33e134e000-33e154e000 ---p 0014e000 fd:00 1998859                        /lib64/libc-2.5.so
33e154e000-33e1552000 r--p 0014e000 fd:00 1998859                        /lib64/libc-2.5.so
33e1552000-33e1553000 rw-p 00152000 fd:00 1998859                        /lib64/libc-2.5.so
33e1553000-33e1558000 rw-p 33e1553000 00:00 0
33e1600000-33e1602000 r-xp 00000000 fd:00 1998863                        /lib64/libdl-2.5.so
33e1602000-33e1802000 ---pAborted

This was not really the outcome I was hoping for, and some quick Googling seemed to indicate that this was a deeper problem than one I wanted to solve.  One of my major concerns is that I’m using a somewhat dated version of Ruby, 1.8.6, caused by our desire to keep our Ruby stack stable across several different CentOS distributions.  However this often is the type of error released in the version after the one I’m currently running…what’s more I suspected strongly that this was an old generate_index gem running up against package metadata that it couldn’t handle properly.  To determine whether this was the problem, or whether it was something else altogether, I decided to try and get the latest and greatest Ruby up and running on a dev box and see if I could index the same repo there.  If I could, it would be worth it to work backwards and figure out if what had changed and perhaps get my Ruby version working with it as well. In a desire to keep as many of the variables the same as possible, I did this on a CentOS 5.5 x86_64 host nearly identical to the Cobbler host, with the exception that Ruby 1.8.6 (and all the goodies that depend on it, including Puppet) were uninstalled after the host was built.  I then grabbed and installed the Ruby 1.9.2 code from http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p180.tar.gz, dropped it on my test host, and did a good ol ./compile; make; sudo make install. This went swimmingly and, as with all things Ruby, faster than expected.  Ruby 1.9 even comes with RubyGems pre-installed…fantastic.  But also as with all things Ruby, it wasn’t as simple as it initially appeared. For a test, I try to get a list of local gems:

# gem list --local
ERROR:  Loading command: list (LoadError)
    no such file to load -- zlib
ERROR:  While executing gem ... (NameError)
    uninitialized constant Gem::Commands::ListCommand

OK, zlib’s missing.  I go and check and sure enough zlib-devel package isn’t installed.  Go ahead and install it, and came across another helpful tip for fixing this without a total rebuild (http://stackoverflow.com/questions/3928965/installing-ruby-on-ubuntu-10-10-using-rvm-problem-with-gem).  So inside the Ruby source tree, I execute:

cd ext/zlib
ruby extconf.rb
make
sudo make install

This is lovely and rebuilds exactly what I need without having to repeat the process.  So now, with everything installed, I cd to the proper directory, execute my index command, and get:

# gem generate_index -d .
ERROR:  While executing gem ... (RuntimeError)
    Gem::Indexer requires that the XML Builder library be installed:
     gem install builder

Oh good!  Well, that’s easy to fix.  Except it wasn’t:

# gem install builder
Successfully installed builder-3.0.0
1 gem installed
Installing ri documentation for builder-3.0.0...
Installing RDoc documentation for builder-3.0.0...

# gem generate_index -d .
ERROR:  While executing gem ... (RuntimeError)
    Gem::Indexer requires that the XML Builder library be installed:
     gem install builder

Hmm.

# gem list

*** LOCAL GEMS ***

builder (3.0.0)
minitest (1.6.0)
rake (0.8.7)
rdoc (2.5.8)

Some more quick Googling (you’ll find this about me…I’m pretty quick to google before investigating.  I feel like less of a geek for it…and I also feel 10x more productive when I don’t waste an entire afternoon poking at something other people figured out already) revealed turned uphttp://stackoverflow.com/questions/4456127/ruby-gem-generate-indexer-broken-with-xml-builder-3-0-0, which indicates it’s a problem with the current XML Builder gem. I decided not to follow the author’s directions in their entirety and rather than hack indexer.rb, just remove the current builder gem and replace with an older one.  To wit:

# gem uninstall builder
Successfully uninstalled builder-3.0.0
# gem install builder -v 2.1.2
Successfully installed builder-2.1.2
1 gem installed
Installing ri documentation for builder-2.1.2...
Installing RDoc documentation for builder-2.1.2...

Once this was done I restarted the indexer, and it appears to be running smoothly.  I do however see a number of the following errors pop up during the processing:

ERROR:  Unable to process /mnt/tmp/gem/gems/FaceToFace-0.1.0.gem
invalid byte sequence in UTF-8 (ArgumentError)

Followed by yet another stack trace.  This very much suggests to me that the problem is what I thought it was — bad metadata in a user-contributed gem that absolutely broke the 1.8 indexer, a problem fixed in 1.9. I’m still waiting for the index to complete, but it’s looking good.  Part II will cover getting this repo into production, and (hopefully) getting this process working on the older Ruby. Some takeaways from this:

  1. RubyGems are fickle, not unlike CPAN.  Running those built as part of a base distro or repository like EPEL where they have been repackaged and tested for your OS is easy…anything else can be unexpectedly difficult
  2. Watch out for unclear cross-dependencies between gems and packages.  This is why it makes sense to use one package system.  If what I have here ultimately has to be the way I get things to work, I will gem2rpm the gems I need, fix them up so they work properly, and install…the danger of breaking gems by, for instance, upgrading zlib-devel won’t alert me as to the problem if I don’t use one management system
  3. Google Is Your Friend.  I could have wasted a lot of time trying to fix the stack trace I initially hit, and I ultimately will if this turns out to be a viable way to mirror RubyGems.  But it was a waste of my time to do so without trying to find a working solution first and determining if the problem was really mine to begin with.  Now I can work backwards to a solution that works within my framework