One of the most tedious, error prone and repetitive tasks involved in SRE and system administration work is the work of building infrastructure. As part of that, it is important that tasks such as image creation, image management, VM provisioning and post build tasks such as OS patching, OS hardening, agent installation and configuration are not only well orchestrated but also engineered correctly.

Enter Infrastructure as code practices. These ensure that we get immutable infrastructure i.e. each time the same product is delivered. While there are plenty of options available to accomplish this task for e.g. we can use a combination of Terraform for provisioning, Ansible for post build and configuration management and Jenkins or Azure DevOps for CI/CD. In this example we shall use Ansible + Jenkisn for the same.

So first lets break down the steps involved

1. Provisioning VMs on both VMware (On-Prem) and AWS environments

2. Performing essential post-build tasks such as Active Directory join

3. Installing critical software agents like Tanium and Splunk

4. Implementing basic security hardening measures

By combining Jenkins' pipeline capabilities with Ansible's flexible playbooks, we'll create a solution that's not only powerful but also adaptable to various environments and requirements.

Prerequisites

Before we dive into the code, make sure you have the following:

• Jenkins server with the Ansible plugin installed

• Ansible control node with access to your VMware and AWS environments

• Necessary credentials for VMware vCenter and AWS

• Basic understanding of Jenkins pipelines and Ansible playbooks

Now, let's explore the solution in detail. We'll start with the Jenkins pipeline script and then break down each Ansible playbook to understand how they work together to create a comprehensive VM provisioning and hardening solution.

Part 1: Groovy Code for Jenkins Orchestration/Pipeline

pipeline {
    agent any

    parameters {
        choice(name: 'ENVIRONMENT', choices: ['vmware', 'aws'], description: 'Select the target environment')
        string(name: 'VM_NAME', defaultValue: 'myvm', description: 'Name of the VM to be created')
        string(name: 'VM_COUNT', defaultValue: '1', description: 'Number of VMs to create')
    }

    stages {
        stage('Prepare Ansible') {
            steps {
                sh 'ansible-galaxy collection install community.vmware community.aws'
            }
        }

        stage('Provision VM') {
            steps {
                script {
                    def ansiblePlaybook = params.ENVIRONMENT == 'vmware' ? 'vmware_provision.yml' : 'aws_provision.yml'

                    ansiblePlaybook(
                        playbook: "${ansiblePlaybook}",
                        inventory: 'inventory/',
                        extras: "-e vm_name=${params.VM_NAME} -e vm_count=${params.VM_COUNT}"
                    )
                }
            }
        }

        stage('Post-Build Tasks') {
            steps {
                ansiblePlaybook(
                    playbook: 'post_build_tasks.yml',
                    inventory: 'inventory/',
                    extras: "-e vm_name=${params.VM_NAME} -e vm_count=${params.VM_COUNT}"
                )
            }
        }

        stage('Security Hardening') {
            steps {
                ansiblePlaybook(
                    playbook: 'security_hardening.yml',
                    inventory: 'inventory/',
                    extras: "-e vm_name=${params.VM_NAME} -e vm_count=${params.VM_COUNT}"
                )
            }
        }
    }
}

Part 2: Ansible playbooks mentioned in the Jenkins pipeline above

1. VMware Provision Playbook

---
- name: Provision VM on VMware
  hosts: localhost
  gather_facts: no
  vars:
    vcenter_hostname: "vcenter.example.com"
    vcenter_username: "administrator@vsphere.local"
    vcenter_password: "{{ vault_vcenter_password }}"
    datacenter_name: "DC1"
    cluster_name: "Cluster1"
    datastore_name: "Datastore1"
    template_name: "ubuntu-20.04-template"

  tasks:
    - name: Create VM from template
      community.vmware.vmware_guest:
        hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_username }}"
        password: "{{ vcenter_password }}"
        validate_certs: no
        name: "{{ vm_name }}-{{ item }}"
        template: "{{ template_name }}"
        datacenter: "{{ datacenter_name }}"
        cluster: "{{ cluster_name }}"
        datastore: "{{ datastore_name }}"
        state: poweredon
        wait_for_ip_address: yes
      loop: "{{ range(1, vm_count|int + 1)|list }}"
      register: vm_facts

    - name: Add VM to inventory
      add_host:
        hostname: "{{ item.instance.ipv4 }}"
        groupname: new_vms
      loop: "{{ vm_facts.results }}"

2. AWS Provision Playbook

---
- name: Provision VM on AWS
  hosts: localhost
  gather_facts: no
  vars:
    region: "us-west-2"
    instance_type: "t2.micro"
    ami_id: "ami-0885b1f6bd170450c"  # Ubuntu 20.04 LTS
    key_name: "my-ssh-key"
    security_group: "default"
    subnet_id: "subnet-12345678"

  tasks:
    - name: Create EC2 instance
      amazon.aws.ec2_instance:
        name: "{{ vm_name }}-{{ item }}"
        instance_type: "{{ instance_type }}"
        image_id: "{{ ami_id }}"
        region: "{{ region }}"
        key_name: "{{ key_name }}"
        security_group: "{{ security_group }}"
        subnet_id: "{{ subnet_id }}"
        wait: yes
        count: 1
      loop: "{{ range(1, vm_count|int + 1)|list }}"
      register: ec2_facts

    - name: Add EC2 instance to inventory
      add_host:
        hostname: "{{ item.instance.public_ip_address }}"
        groupname: new_vms
      loop: "{{ ec2_facts.results }}"

3. Post-Build Tasks Playbook

---
- name: Perform post-build tasks
  hosts: new_vms
  become: yes
  vars:
    ad_domain: "piyops.com"
    ad_username: "admin"
    ad_password: "{{ vault_ad_password }}"

  tasks:
    - name: Join Active Directory
      community.windows.win_domain_membership:
        dns_domain_name: "{{ ad_domain }}"
        domain_admin_user: "{{ ad_username }}"
        domain_admin_password: "{{ ad_password }}"
        state: domain

    - name: Install Tanium agent
      ansible.builtin.package:
        name: tanium-agent
        state: present

    - name: Install Splunk agent
      ansible.builtin.package:
        name: splunkforwarder
        state: present

    - name: Configure Splunk agent
      ansible.builtin.template:
        src: splunk_agent_config.j2
        dest: /opt/splunkforwarder/etc/system/local/inputs.conf
      notify: Restart Splunk agent

  handlers:
    - name: Restart Splunk agent
      ansible.builtin.service:
        name: splunkforwarder
        state: restarted

4. Security Hardening Playbook

---
- name: Perform basic security hardening
  hosts: new_vms
  become: yes

  tasks:
    - name: Update all packages
      ansible.builtin.package:
        name: "*"
        state: latest

    - name: Ensure firewall is installed and enabled
      ansible.builtin.package:
        name: ufw
        state: present

    - name: Enable firewall
      community.general.ufw:
        state: enabled
        policy: deny

    - name: Allow SSH
      community.general.ufw:
        rule: allow
        port: '22'

    - name: Disable root login via SSH
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^PermitRootLogin'
        line: 'PermitRootLogin no'
      notify: Restart SSH

    - name: Set strong password policy
      ansible.builtin.lineinfile:
        path: /etc/security/pwquality.conf
        regexp: '^{{ item.key }}'
        line: '{{ item.key }} = {{ item.value }}'
      loop:
        - { key: 'minlen', value: '14' }
        - { key: 'dcredit', value: '-1' }
        - { key: 'ucredit', value: '-1' }
        - { key: 'ocredit', value: '-1' }
        - { key: 'lcredit', value: '-1' }

  handlers:
    - name: Restart SSH
      ansible.builtin.service:
        name: sshd
        state: restarted

Assumptions and notes

1. The Jenkins pipeline assumes you have the Ansible plugin installed and configured.

2. VMware and AWS credentials are stored securely (e.g., using Jenkins Credentials or Ansible Vault). Read about how to configure and store credentials in Ansible Vault in this Ansible Vault Tuturial.

3. The VMware playbook assumes you have a template named "ubuntu-20.04-template". Replace this with your own image/template.

4. The AWS playbook uses a specific AMI ID for Ubuntu 20.04 LTS in the us-west-2 region. You may need to adjust this for your region.

5. The post-build tasks assume you have repositories configured for Tanium and Splunk agents.

6. Active Directory join is performed using the community.windows.win_domain_membership module, which may require additional setup for Linux systems.

7. Basic security hardening includes package updates, firewall configuration, SSH hardening, and password policy settings.

Additional points to keep in mind

1. Create the necessary Ansible playbooks and place them in your Jenkins workspace.

2. Adjust the variables and tasks in the playbooks to match your specific environment and requirements.

3. Set up the required credentials in Jenkins for VMware, AWS, and any other necessary systems.

4. Create a new Jenkins pipeline job and use the provided pipeline script.

5. As a best practise, remember to always store your code on a code repository like Github or Azure Repo for better manageability, version control, security and backup.

Remember to review and adjust the assumptions and configurations to match your specific environment and security requirements. Making notes, documentation and keep adjusting your code to

Hope this was helpful and gives a broad direction for you to get started in your SRE / DevOps / IaC journey.