All Things Open 2016: Additional Jenkins Comments (Part 1)

At the end of October, I gave a talk about my day job at a little conference called All Things Open. I LOVE sharing what I do, especially since I can now.  Not like I ever couldn't in the 3 letter agency way of "I work with computers". The 3 letter company I previously worked for was commercial. Not everyone gets excited about autoconfiguring rack architecture. Except maybe the Open Stack people.

Jenkins, the open source automation application, is amazing. The community is incredibly active, and it seems as if I'm constantly learning some new plugin or trick to make my pipelines go. Because the best thoughts/suggestions/witty retorts are always the ones you have while you're attending the next session in the conference, I wanted to share some of them with whoever happens to come across this space. Because I have several thoughts, I'm spreading out the love over a few posts.

First things first: repo + slides are available up on GitHub. That app is so basic... It also made me realize the Javascript/Node world is pretty much Professor Farnsworth's nanobot world. 99% guarantee it's using the embers of the old hotness.

The sample project is great for demonstrating the function of MultiBranch Pipeline, the final form of pipeline management. Sidebar: The ultimate form, automated management of an entire organization of multibranch pipelines, can only be reached by GitHub Organizations and BitBucket Teams. If you're starting from "I just want to lay down some test groovy", start with the "Pipeline" job.  A field of possibilities awaits you there

There are effectively no limits to what you can do with your Pipeline jobs. In my personal development, I use the inline script to test out ideas and quickly get going. I'll use the "from a file in your repository" option as my more permanent pipeline. I like versioning my pipeline, and have it update right after I push new code without having the extra step of logging into Jenkins to update the job.

From this...

...to this

Working on SCM managed code is, I would argue, necessary when there are many people going in and out of a common job. In release, we not only develop Jenkins but we're heavy users too. We've been using pipelines since they came to be, and we manage the various scripts through essentially a library. The pipeline jobs in Jenkins look similar, just they use a different "flow" for actions.  This gives us all a chance to see exactly what's happening in each of our flows, do code reviews, handle versioning, backup for all the things, preventing wild wild west job modification, and pretty much any other benefits from using scm.

Some repos require digging through the layers of Troy: different mix of pipeline abilities in older scripts. We're already starting to convert to using Jenkinsfiles for running our jobs, translating the extremely common, yet subtlety customized, code flow into individual repos. This also keeps the associated helper scripts and additional tweaks with the job. The common scripts are now put in a library accessible from the individual jobs.

Of course, not everyone wants or should know the details of what happens in your jobs. That's probably the main driving force for the question above In that case, then this works too.  The peeps with teh powa admins can control whatever they need to in their repository, and jobs can chug along without others having to see how the sausage (or more appropriately for the Sidebar: beer) gets made.  Sidebar: beer pairs well with pipelines.

This longer ramble answers one question that came up during the Q&A: can I set up one repo with a common build script and use it with a combination of helper scripts to manage several different similar tasks. Yes, yes you can. We're doing it today. There are a few suggestions I'd like to make if you go that route, especially since there are so many features that didn't exist before:

• You potentially lose the benefits of managing that common job within your scm. Changes must be applied within Jenkins itself to all your jobs. And that's a pain. </understatement>
• CloudBees offers a Templates Plugin that really solves this problem. Check within your company to see if you have a CloudBees subscription as that will really shorten your configuration time.
• You could possibly get around your helper script by using variables that just tell your main job which helper script to load. Make sure you have good security set up before going down this route.

I'll be experimenting with some more of these suggestions as part of just further Jenkins discovery, and will share what I find! Feel free to ask questions, and share how you've used Jenkins pipeline. I'll do some other posts with additional thoughts and reflections so stay tuned.

</sidebar>

<sidebar>
I'm trying to consciously only add 1 space between typed sentences because along that front the Manual of Style put up another "W" for Chicago (no pressure Bears). It's so hard after living double spaced 4-lyfe. I think it looks cramped as if everything is one run on sentence in a poem that never found its way. But hey, the people who get passionate about typography say 1 space. However, you'll pry semicolons out of my cold, dead hands.
</sidebar>

Converting from HudsonTestCase to JenkinsRule

<speedrack alt_text="Getting your plugins up to speed">
</speedrack>

Let's be real: testing is usually not the first thing on software developer's minds when creating a new system. I may or may not have been involved in a project or 2 like that in my day.  And the "test driven development" that we tried to learn in the early days of programming didn't really prove much other than driving home certain coding styles.  Curse you WebCat.... Not putting in a good word for your fellow felines.

However; as we get older, we learn from our mistakes.  Namely the ones found in debugging.  If you're reading this, you've already gotten to the wiser part, and jumped headfirst/feetfirst/behindfirst/bellyfloped into Jenkins, throwing some abandon in when taking up custom plugin development.  You've also decided to go feet first since you're looking to test what you use for tests.  Bonus points for keeping your plugin up to date.  The way not to let the defect tide crash over your head is to get that core version up.  My fellow crew, change your dependency to the plugin parent pom, and get ready for overall plugin improvements.

The tide of progress hides some interesting sandbars, and the first one you hit while paddling out of the shallows is Junit4.  While I'm about ready to keep rolling in the deep with this ocean metaphor, I'm going to stay on shore and just let the cruse control of the new @Rule handle things for me.

Hot: JenkinsRule
Not: HudsonTestCase

Using the amazing powers of JenkinsRule, we are able to replace a lot of things one to one.  I've included a good list of changes in the following gist.  Check out the official Jenkins docs for extra unit test tips and tricks.

Along those lines, I found a few other snags that might need some extra explaining.  Since the tests aren't a subclass of HudsonTestCase, you won't have the benefits of being a RootAction.  But it's not an issue since JenkinsRule throws in this functionality too.  Simply create a subclass of JenkinsRule, and customize away.  Don't forget to move your resource into a subfolder for your custom JenkinsRule; check out this.  Ex: /test/resources/com/example/UnitTest/hello.jelly -> /test/resources/com/example/UnitTest/CustomRule/hello.jelly

So while we definitely are out there running tests, acting all carefree, looking at the green checkmarks in GitHub, there's one last struggle you might hit in your tests: using a security realm.  MockAuthorizationStrategy combined with setting JenkinsRule.jenkins.setAuthorizationStrategy(), you can easily throw together a whole system of permission for users.  One of the best examples of this is cloudbees-folder-plugin/FolderTest.

Hopefully this helps with potential snags that you might run into when writing tests for your test framework.  See you on the friendly IRC/Google Groups/GitHub PRs!

</sidebar>

Finding Files in Linux Based on Known Time

I've slowly become a command line convert even to the point of finding it easier to just straight code in vi rather than download each to my box.  There's so much control over everything you do & you're uninhibited by what you can find within your program files.

However one thing I think that keeps people around in the wonderful world of GUIs is the ability to see files in a tree structure.  It's very visual, we all can process the information incredibly quickly.  And this is incredibly useful for utilizing what we do best which is make sense of unstructured data.  & yes I know it's a stretch to call a grouping of pictures with clearly defined labels "unstructured".  However, like creating the super long specialized view for looking at an entry in the database, sometimes it's just easier to take the simple search and then scan for what you need, especially if it's a one time use case.  This was before I discovered that burred within the man pages, "find" is incredibly powerful.

Using find to search a given time range

So I needed to find a file that was last modified between 5am and 5:10am on my system.  I had no idea what the file would be, but I figured that I should get the list of all the files and narrow from there.  The immediate issue you'll notice with find is that everything is based from now.  Which is great if you're searching generally or have the exact minute difference from where you were today and the time your file exists.  Having to constantly update that would defeat the intent of making life easy.

The solution comes with -newerXY an option that allows you to look for newer files matching the parameters that are put in X, Y and as the argument.  Since we have a time and not a particular file we're going to specify Y = t.  X can be used to specify time of creation, modification, or access.

For example to find all files in root that have been modified between 5:00am and 5:10am on 28 March:
find / -newermt '28 March 2013 5:00' -not -newermt '28 March 2013 5:10'

And this data become even more useful when able to sort and utilize the powers of ls.  The magic of xargs allows us in this case to pipe the results of this query as the "file system" to list and sort.

For example to nicely format the above files with timestamps and in descending order:
find / -newermt '28 March 2013 5:00' -not -newermt '28 March 2013 5:10' | xargs ls -t -l
 
Piping even further, you can refine the search for the file names or even more though grep.   But that's up to your own personal application.  You can keep piping forever Mario, but the princess is still in another castle!


Other bars to sample around this problem:
Virtuelvis
Stack Overflow -> this is the place to go for all your programming woes
Super User -> similar to Stack Overflow
Cyber Citi xargs

</sidebar>

Convert Port Numbers to Alias: BNT Style

I'm going to come right out and say it: networking is really really really picky.  I've seen 5 year olds with a more diverse palette than how 2 switches need to be tuned in order to communicate.  I'll try to post some tips about IBM's BNT switches in general  because there are some funky-tastic configuration settings.  And getting it to pair well with Cisco switches is something that can be fine tuning with a dial where a touch is all it takes to calibrate.  Also there is so little out there for BNT; I've spent so much time traversing the pathways of the internets trying so hard to find something a bit more than the manual.  You can find more dust in a clean room.

There's a speck.  It's on that chair
<http://www.wiinc.net/images/cleanroom.JPG>

IBM (through BNT) has several different switches; one is a Chassis Switch, nicknamed "Compass" switch that allows connection into a chassis unit on a rack.  You really don't hit things like this unless you're in an enterprise system; however, some person who was really on top of their game made sure that the port conventions between switches was the same!  Bonus!

For things like confirmation and automation, the important thing to check is running-config.  From this mode you can see all the ports on the associated vlans, but the ports are only shown in their alias form.  It really annoys me that not only are the ports only displayed in alias so it's pretty difficult to do automation.  Also, the ports merge in a very weird way.  So the best way I've found to solve this problem is to make assumptions about inputs: everything is already sorted in the right order.  The script is also in groovy.

/**
* Takes a list of ports in a String form and converts them to the compass
* port. The String can be one or more ports depending on what is passed in.
*
* ex: 1,3-42,60 -> INTA,INTA3-INTC14,EXT18
* @param ports
* @return String the Compass switch aliases of the ports
*/
static _convertToAlias(String ports){
def commaSplit = ports.split(/,/)
def all = ""
def numsToLetters = ["A", "B", "C"]

commaSplit.each { piece ->
if(piece.isInteger()){ //singular
def working = piece.toInteger()
def letter = (working/14 > 3) ? "EXT" : "INT" + numsToLetters[(working%14 == 0 ? (working/14 -1) : (working/14) as int)]
def number = (working/14 <= 3) ? ((working%14 != 0) ? working%14 : 14) : working - (14*3)
all += (letter+number) + ','
} else { //range
def pieces = piece.split(/-/)
all += _convertToAlias(pieces[0]) + "-" + _convertToAlias(pieces[1])+ ","
}
}

return all[0..-2]
}


One incredibly effective way to use this code is to combine it with the other method that I wrote earlier to combine ranges of integers.  A few steps closer to automation!  The future isn't as exciting as the 70s thought it would be, it's more because you can't tell me that watching the computer configure itself isn't the future.  *cue epic music*

</sidebar>