The following assumptions are made: You are familiar with Amazon Web Services enough to understand the first step in the Jenkins Master on AWS section.

Jenkins Master on AWS

Configure your AWS account
I like to keep AWS from using the default for options when setting up my EC2s. It keeps me aware of the layout of my infrastructure without creating too much cruft.

  1. Configure a Virtual Private Cloud with an Internet Gateway and a Public Subnet. Ensure your Public Subnet is configured to auto-assign IP addresses. It’s optional but recommended to also create an Elastic IP to associate with your Jenkins EC2 (either via Network Interface or directly with the EC2 instance).

  2. Create an EC2 instance within the infrastructure created in step #1. I used a Centos 7 Amazon Machine Instance (AMI) entitled "CentOS 7 (x86_64) - with Updates HVM)". This AMI is free-tier eligible, and a t2.micro is large enough to run a Jenkins Master server. Obviously, scale up if need be. No need to assign an IAM role to this server. TODO is storage actually required?

Note the key pair used to access the instance. We will use this key pair later to allow the Jenkins Master to access the agents. I named my key pair "jenkins" 'cause I’m creative like that :-) .
The EC2 instance created in step #2 will be referred to as the "Jenkins Master" going forward. Technically, this makes sense because really all this server does is run the Jenkins Master process.
Be sure the Route Table associated with your VPC has an entry for the Internet Gateway. This does not happen automatically when associating an Internet Gateway to a VPC.
Install Jenkins

  1. Log into the Jenkins Master EC2 instance using your jenkins key pair.

  2. Follow the instructions to install Jenkins. If using the same Centos 7 AMI I did, Red Hat Distributions Install instructions here.

For me, installing Jenkins (latest LTS) boiled down to these commands, then logging in using the password when first visiting Jenkins Master at port 8080:

sudo yum install java
sudo yum install wget
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins
sudo service jenkins start
sudo cat /var/lib/jenkins/secrets/initialAdminPassword #To get the initial admin password
Be sure a version of Jenkins greater than 2 (major version) installs. So 2.32.1-1.1 works. We will use Jenkins Plugins that require a Jenkins major version of greater than or equal to 2.
Jenkins provides a nice startup interface for installing plugins, but all these plugins can be installed from the Jenkin’s Update Center. Since we don’t need to install many plugins for this tutorial we will move past this (click the X in the top right), but it’s useful to know what plugins are offered and what plugins are "suggested".

Jenkins Master creates Agents on AWS

Install and Configure the Amazon EC2 Plugin
I will keep this brief by not attempting to describe Jenkins interface in too much detail. That being said, use the second bar from the top (not the first black one, but the light blue one below that) to orient yourself; It will dynamically populate as you navigate through Jenkins.
Prudent to set up an identifiable Jenkins user with the same priviledges as admin for traceability purposes on who did what instead of doing everything as admin. Create your new user through "Manage Jenkins" > "Manage Users" and ensure your user has admin privileges in "Manage Jenkins" > "Configure Global Security".

  1. From the home screen go to "Manage Jenkins" > "Manage Plugins". Install the Amazon EC2 plugin. It’s dependencies will install with it.

  2. Navigate to "Manage Jenkins" > "Configure System". The Amazon EC2 plugin added a "Cloud" section. Click "Add a new cloud" > "Amazon EC2". Several configuration settings will appear; Basically now we populate these settings!

  3. Follow the instructions below to Connect to your Amazon EC2 via a Jenkins IAM Role

  4. Click the "Add" button next to AMIs setting. This will give us some AMI settings to populate.

  5. Choose an AMI to use. Populate corresponding settings about the AMI and how you would like it spun up. These settings include: AMI ID, Instance Type, EBS Optimized, Availability Zone, Security Group Names, AMI Type, and Use Spot Instance. For me, I opted for the same instance type for my Build Agents as my Jenkins Master, a Centos 7 (AMI) called "CentOS 7 (x86_64) - with Updates HVM)".

  6. For remote user, check what the default user is one should log in with when SSHing into the AMI. For me, it’s "centos". TODO does Remote FS Root & Root command prefix & Remote SSH port & Labels matter?

  7. Use the Init Script setting to configure the AMI you choose to handle Jenkins, adjust security as necessary, and add the RSA public key to the authorized key file on the server. This last step will allow the Jenkins Master to log into the Build Agent spun up as needed. Since the AMI I chose was very bare bones my Init Script looked like this:

echo "=== 0 of 2 | Jenkins Master Init Script ==="
echo "=== 1 of 2 | Configuring Build Agent ==="
jenkinsRSApub="[Your key pair's public key without brackets in quotes]"
echo $jenkinsRSApub >> /home/centos/.ssh/authorized_keys
setenforce 0
echo "=== 2 of 2 | Installing Required Software ==="
sudo yum update --assumeyes
sudo yum install wget --assumeyes
sudo yum install scp --assumeyes
sudo yum install java-1.8.0-openjdk --assumeyes ###java instead of java-1.8.0
sudo yum install git --assumeyes

8.Note the "Advanced" button on the far-right at the bottom of the AMI settings. This button can be difficult to see depending on how the settings populate. Click it to see some additional AMI configurations. I set the "Number of Executors" and "Instance Cap" to 1 to keep several pipelines from running at once on a Build Agent and restrict Jenkins from spinning up several Build Agents to process jobs, added the Subnet ID for my VPC, added my ARN for my IAM instance profile, and checked "Connect By SSH Process". 9.Now you are ready to test that Jenkins Master creates Build Agents using EC2s as needed.

Connect to your Amazon EC2 via a Jenkins IAM Role

  1. Fortunately, the Amazon EC2 plugin has an "IAM setup" section. There, it provides the JSON to properly configure the Jenkins IAM Policy the Jenkins IAM Role would use. I added the iam:PassRole permission to my policy. TODO Find out if things work the same without the iam:PassRole permission in the policy.

  2. Navigate to your Identity and Access Management (IAM) section of your AWS account and create a Jenkins Policy using the JSON provided plus iam:PassRole.

  3. Create a Jenkins Role that uses the Jenkins Policy. Save the following details when they are made available to you: "Access Key ID", "Secret Access Key", and "IAM Role to Use" (an Amazon Resource Name (i.e. ARN, example: "arn:aws:iam::123456789012:role/MyIAMRoleName") )

  4. Once you have these details navigate back to Jenkins > Manage Jenkins > Configure System, Cloud section. For the "Amazon EC2 Credentials" setting click the "Add" dropdown and select "Amazon EC2". A popup will appear with credential configuration settings. For "Kind" select "AWS Credentials". Populate the "Access Key ID", "Secret Access Key", and "IAM Role To Use". "IAM Role To Use" needs to be an ARN. Then complete the form by clicking "Add"; Not having all the fields filled out does not cause an issue. TODO might not need the "IAM Role To Use" ARN if Jenkins user already exists.

  5. Copy the private key pair (my "jenkins" key pair) into the "EC2 Key Pair’s Private Key" setting.

  6. Select a Region to use. It has to be a Region the key pair from step #5 is registered to.

  7. Click "Test Connection". It should say "Success" in small black text or display an error in bold red text.

Test Jenkins Master Creates Build Agents Using EC2s As Needed

  1. On the Jenkins Homepage, navigate to "New Item".

  2. You will be asked to create a name for this item, then select what type of item it is. In my case at this point I can only select "Freestyle Project" and press "OK". 3.

Jenkins Master uses Agents to Process Pipelines on AWS

Configure Amazon EC2 Plugin

Build a Pipeline Using a Jenkinsfile

Configure Jenkins to Trigger Pipelines from Github Activity

Going Forward

TODO: .build agents for specific repos .Automatically kick off build when PR is opened into master

Spread the word