{"id":347,"date":"2023-01-05T01:43:00","date_gmt":"2023-01-05T01:43:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=347"},"modified":"2023-01-05T01:43:00","modified_gmt":"2023-01-05T01:43:00","slug":"packer-tutorial-for-beginners","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=347","title":{"rendered":"How to Connect to Cloudways Server [SSH & SFTP]"},"content":{"rendered":"

Packer is an open-source VM image creation tool <\/strong>from Hashicorp<\/a>. It helps you automate the process of Virtual machine image creation on the cloud and on-prem virtualized environments.<\/p>\n

To put it simply, whatever manual steps you perform to create a Virtual machine image can be automated through a simple Packer config template<\/strong>. You declare the state of the VM image you need, and Packer builds it for you.<\/p>\n

Packer Use Cases<\/h2>\n

The following are the main use cases for Packer.<\/p>\n

    \n
  1. Golden Image Creation:<\/strong> With packer, you can template the configurations required for a golden VM image that can be used across organizations.<\/li>\n
  2. Monthly VM Patching: <\/strong>You can integrate Packer into your monthly VM image patching pipeline.<\/li>\n
  3. Immutable Infrastructure:<\/strong> If you want to create an immutable infrastructure <\/a>using VM images as a deployable artifact, you can use Packer in your CI\/CD pipeline.<\/li>\n<\/ol>\n

    Basically, all the above use cases are part of the CI pipelines or IaaC code pipelines<\/strong> where every Packer template is version controlled. Even the packer template development or update<\/strong> should go through all the CI tests before getting deployed in a project environment.<\/p>\n

    Packer Tutorial For Beginners<\/h2>\n

    In this Packer beginner tutorial, you will learn the following.<\/p>\n

      \n
    1. The basic building blocks of Packer.<\/li>\n
    2. Packer Workflow<\/li>\n
    3. Installing Packer<\/li>\n
    4. Understanding Packer HCL template.<\/li>\n
    5. Creating an AWS AMI using Packer<\/li>\n<\/ol>\n

      Let’s get started with the Packer setup.<\/p>\n

      Install Packer<\/h2>\n

      You can have Packer installed on your local workstation or on a cloud instance. In the actual project, the Packer installation would be part of the Jenkins<\/a> agent or any tooling which is part of the CI\/CD process.<\/p>\n

      \n
      Note:<\/strong><\/b> If you are setting up Packer on your workstation for debugging or testing purposes, you need the AWS access and secret keys for Packer to interact with AWS services. Make sure you have the AWS access keys set on the file ~\/.aws\/credentials<\/code>. It is not mandatory to have a credentials file. You can pass the AWS credentials using the packer variable declaration as well.<\/div>\n<\/div>\n

      Step 1:<\/strong> Download the latest Packer executable from the Packer downloads page. https:\/\/www.packer.io\/downloads.html<\/a>. It is available for Windows, Linux, and other Unix platforms.<\/p>\n

      \"packer<\/figure>\n

      For Linux, you can install Packer from the respective package managers or you can get the download link from the download button and download it using wget<\/code>.<\/p>\n

      For example, In Ubuntu<\/p>\n

      wget -O- https:\/\/apt.releases.hashicorp.com\/gpg | gpg --dearmor | sudo tee \/usr\/share\/keyrings\/hashicorp-archive-keyring.gpg\necho \"deb [signed-by=\/usr\/share\/keyrings\/hashicorp-archive-keyring.gpg] https:\/\/apt.releases.hashicorp.com $(lsb_release -cs) main\" | sudo tee \/etc\/apt\/sources.list.d\/hashicorp.list\nsudo apt update && sudo apt install packer<\/code><\/pre>\n
      Step 2:<\/strong> Verify packer installation by executing the packer command.<\/p>\n
      packer version<\/code><\/pre>\n
      You should see the output as shown below.<\/p>\n
      \"Verifying<\/figure>\n
      Building VM Image (AWS AMI) Using Packer<\/h2>\n
      Building a VM image using Packer is a simple process. The workflow is somewhat similar to building VMs using Vagrant<\/a><\/p>\n
      Take a look at the following high-level Packer workflow diagram.<\/p>\n
      \"\"<\/figure>\n
      Let’s break down the process.<\/p>\n
        \n
      1. Declare all the required VM configurations in an HCL (Hashicorp configuration language) or a JSON file. Let’s call it the Packer template<\/strong>.<\/li>\n
      2. To build the VM image, execute Packer with the Packer template..<\/li>\n
      3. Packer authenticates the remote cloud provider and launches a server. If you execute Packer from a cloud environment, it leverages the cloud service account for authentication.<\/li>\n
      4. Packer takes a remote connection to the server (SSH or Winrm).<\/li>\n
      5. Then it configures the server based on the provisioner you specified in the Packer template (Shell script, Ansible, Chef, etc).<\/li>\n
      6. Registers the AMI<\/li>\n
      7. Deletes the running instance.<\/li>\n<\/ol>\n
        Packer Template Reference<\/h2>\n
        All the code used in this article is hosted on a Packer Template Github repository.<\/a> You can clone or fork it to keep it as a reference. Or you can use the template directly from the packer-aws-shell<\/code> folder<\/p>\n
        git clone https:\/\/github.com\/techiescamp\/packer-templates<\/code><\/pre>\n
        Please feel free to contribute to the repo with more templates.<\/p>\n
        Packer HCL Template Explained<\/h2>\n
        \n
        Note:<\/strong><\/b> From version v1.7<\/code> Packer uses HCL format templates. JSON support is also there. But it’s recommended to change to HCL format as the newly developed features will not be added to the JSON-based template. If your exiting project or organization users JSON templates, you can convert them to HCL templates. See Upgrading JSON to HCL<\/a> for more information.<\/div>\n<\/div>\n
        Let’s have a look at the HCL Packer template in detail.<\/p>\n
        Here is the high-level packer template structure.<\/p>\n
        \"Packer<\/figure>\n
        An HCL template has a few blocks. Let’s understand each block with sample configuration<\/strong> before we create the AMI with Packer<\/p>\n
        Variables<\/h3>\n
        Using the variable block, you can declare the default Packer parameters as variables. The declared variables can be overridden from the command line during the packer build process.<\/p>\n
        You can make use of packer variables for dynamic configuration when used with CI tools like Jenkins.<\/p>\n
        Let\u2019s understand different scenarios for variable usage.<\/p>\n
        Using Variable Within Template<\/h4>\n
        The variables<\/code> block holds all the default variables within a template. An example is shown below.<\/p>\n
        variable \"ami_id\" {\n  type    = string\n  default = \"ami-6f68cf0f\"\n}<\/code><\/pre>\n
        The declared variables can be accessed in other parts of the template using \"${var.app_name}\"<\/code> syntax. An example is shown below.<\/p>\n
        source_ami    = \"${var.ami_id}\"<\/code><\/pre>\n
        Using Environment Variables in Templates<\/h4>\n
        Packer lets you use the system environment variables. First, you need to declare the environment variables in the variable section to use them in the other parts of the template.<\/p>\n
        Let’s say, you want to use the SCRIPT_PATH<\/code> environment variable that holds the path to a shell script that has to be used in the shell provisioner. You can declare that variable as shown below.<\/p>\n
        variable \"script_path\" {\n  default = env(\"SCRIPT_PATH\")\n}<\/code><\/pre>\n
        Then, you can use the script_path<\/code> variable in the provisioned block like shown below.<\/p>\n
        provisioner \"shell\" {\n    script = \"${var.script_path}\/demo-script.sh\"\n  }<\/code><\/pre>\n
        Using Command Line Variables<\/h4>\n
        All the variables declared in the variable block can be overridden during the Packer run time using -var<\/code> a flag followed by the variable value.<\/p>\n
        For example,<\/p>\n
        packer build -var 'app_name=httpd' ami.pkr.hcl<\/code><\/pre>\n
        \n
        Note:<\/strong><\/b> In the above example, ami.pkr.hcl<\/code> is the packer template file.<\/div>\n<\/div>\n
        Using an HCL file<\/h4>\n
        You can also use an HCL file to eternalize all the variables in the template. Let’s say we have a variable file named  vars.packer.hcl<\/code>, you can add the variables as shown below.<\/p>\n
        app_name = \"httpd\"\nami_id   = \"ami-6f68cf0f\"<\/code><\/pre>\n
        And during the packer build, you can pass this file as shown below.<\/p>\n
        packer build -var-file=\"vars.packer.hcl\"<\/code><\/pre>\n
        Using a JSON File<\/h4>\n
        You can also list all the variables in a JSON file and pass it during the packer build as shown below.<\/p>\n
        packer build -var-file=variables.json ami.pkr.hcl<\/code><\/pre>\n
        locals<\/h3>\n
        Locals are also variables, but they cannot be overridden. It is scoped within the Packer template.<\/p>\n
        locals {\n    app_name = \"httpd\"\n}<\/code><\/pre>\n
        You can interpolate a variable with a local as shown below.<\/p>\n
        locals {\n  ami_name = \"${var.app_name}-prodcution\"\n}<\/code><\/pre>\n
        If you want to pass any sensitive information in a variable<\/strong>, you can convert it to a local value and then mark it as sensitive. This way, the value won’t be printed anywhere in the Packer build logs.<\/p>\n
        local \"secret_key\" {\n  key = \"${var.secret_key}\"\n  sensitive  = true\n}<\/code><\/pre>\n
        Source<\/h3>\n
        Packer works on the concept of the builder plugin. For every cloud provider, there are specific builder plugins that interact with the Cloud provider API to create the images.<\/p>\n
        In this block, we mention the builder plugin details. For example, if you want to build AWS AMI, you can specify amazon-ebs<\/code> as  a builder plugin. You have to provide all the mandatory parameters associated with the amazon-ebs<\/code> builder plugin.<\/p>\n
        Also, In the following code block, you can see that we are calling ${var.ami_id}<\/code> declared in the variable block.<\/p>\n
        Check the official builder plugin documentation <\/a>to know about the list of supported builders.<\/p>\n
        source \"amazon-ebs\" \"httpd\" {\n  ami_name      = \"PACKER-DEMO-${local.app_name}\"\n  instance_type = \"t2.micro\"\n  region        = \"eu-west-1\"\n  source_ami    = \"${var.ami_id}\"\n  ssh_username  = \"ec2-user\"\n  tags = {\n    Env  = \"DEMO\"\n    Name = \"PACKER-DEMO-${var.app_name}\"\n  }\n}<\/code><\/pre>\n
        build<\/h3>\n
        In this block, we declare all the provisioners and post-processor.<\/p>\n
        build {\n  sources = [\"source.amazon-ebs.httpd\"]\n\n  provisioner \"shell\" {\n    script = \"script.sh\"\n  }\n}\n<\/code><\/pre>\n
        Provisioner<\/h3>\n
        The provisioner block is part of the build<\/code> block. Here you can mention how you want to configure the VM image. You can run shell scripts, Ansible playbooks, Check cookbooks<\/strong>, etc. See the provisioners page<\/a> for all the supported provisioners.<\/p>\n
        The script or ansible-playbook <\/code>should be present in the folder where you have the Packer template.<\/p>\n
        provisioner \"shell\" {\n    script = \"script.sh\"\n  }<\/code><\/pre>\n
        Also, there is a file provisioner using which you can copy files from the packer machine to the AMI.<\/p>\n
        For example,<\/p>\n
        provisioner \"file\"{\n  source = \"\/build\/artifact\/code.jar\"\n  destination = \"\/app\/code.jar\"\n}<\/code><\/pre>\n
        Post-Processor<\/h3>\n
        Post-Processor is not a mandatory block. In this block, we can specify what to do after creating the Image. For example, you can execute a local shell script to create a file with all the image details.<\/p>\n
        post-processor \"shell-local\" {\n    inline = [\"echo foo\"]\n  }<\/code><\/pre>\n
        Also, you can use the manifest post-processor to get the AMI details in the manifest.json<\/code><\/strong> file.<\/p>\n
          post-processor \"manifest\" {\n    output     = \"manifest.json\"\n    strip_path = true\n  }<\/code><\/pre>\n
        Creating AWS AMI Using Packer HCL Template<\/h2>\n
        Now let’s see how to create an AMI using the Packer HCL template.<\/p>\n
        \n
        Note:<\/strong><\/b> I have given a basic example with minimal configurations just for your understanding. When it comes to Packer implementation in the real-time project, more customizations may be needed in terms of variables, naming schemes, builder plugging, etc.<\/div>\n<\/div>\n
        Here is what the template does.<\/p>\n
          \n
        1. Packer deploys an ec2 RHEL instance<\/strong> based on AMI id ami-01e78c5619c5e68b4 <\/code>in the eu-west-1<\/code> region.<\/li>\n
        2. Executes the shell script<\/strong> on the RHEL instance from the script.sh<\/code> script provided in the template directory<\/li>\n
        3. Registers the AMI<\/strong> with the name given in the template.<\/li>\n
        4. Deletes the running machine.<\/li>\n<\/ol>\n
          Let’s get our hands dirty with a practical example.<\/p>\n
          Step 1:<\/strong> Create a folder named “packer-vm<\/code>“.<\/p>\n
          mkdir packer-vm<\/code><\/pre>\n
          Step 2:<\/strong> Inside the packer-vm<\/code> folder, create a file named “vm.pkr.hcl<\/code>” and copy the following HCL template.<\/p>\n
          \n
          If we put together all the blocks I explained in the above section, we get the following HCL Template.<\/div>\n<\/div>\n
          \nvariable \"ami_id\" {\n  type    = string\n  default = \"ami-01e78c5619c5e68b4\"\n}\n\nvariable \"app_name\" {\n  type    = string\n  default = \"httpd\"\n}\n\nlocals {\n    app_name = \"httpd\"\n}\n\nsource \"amazon-ebs\" \"httpd\" {\n  ami_name      = \"PACKER-DEMO-${local.app_name}\"\n  instance_type = \"t2.micro\"\n  region        = \"us-west-2\"\n  source_ami    = \"${var.ami_id}\"\n  ssh_username  = \"ec2-user\"\n  tags = {\n    Env  = \"DEMO\"\n    Name = \"PACKER-DEMO-${var.app_name}\"\n  }\n}\n\nbuild {\n  sources = [\"source.amazon-ebs.httpd\"]\n\n  provisioner \"shell\" {\n    script = \"script\/script.sh\"\n  }\n\n  post-processor \"shell-local\" {\n    inline = [\"echo foo\"]\n  }\n}<\/code><\/pre>\n
          Step 3: <\/strong>Create a folder named script<\/code> inside the packer-vm<\/code> folder and cd into the script directory.<\/p>\n
          mkdir script && cd script<\/code><\/pre>\n
          Step 3:<\/strong> Create a file name script.sh<\/code> and copy the following shell script to it.  It just updates the yum repo and installs httpd<\/code>. Here, you can add all the shell scripts that you want to get executed during the AMI creation.<\/p>\n
          #!\/bin\/bash\nsudo yum -y update\nsudo yum install -y httpd<\/code><\/pre>\n
          Step 4: <\/strong>[Very Important<\/strong>] Now that we have all the required files, let’s understand the required packer authentication.<\/p>\n
          If you are running Packer on your workstation for debugging or testing purposes, you need the AWS access and secret keys for Packer to interact with AWS services<\/strong>. Make sure you have the AWS access keys set on the file ~\/.aws\/credentials<\/code>.<\/p>\n
          Alternatively, you can pass AWS keys in the source block, as shown below. It is not a recommended practice<\/strong>. Accidently you might commit this to a source code repo.<\/p>\n
          source \"amazon-ebs\" \"example\" {\n  access_key = \"ASDFASDFASDFJQWEKONNOASDF\"\n  secret_key = \"SDFQ324876134HJKLASDBJKASDJFHBAJSDFJKHASKDG\"\n}<\/code><\/pre>\n
          If you are running Packer from an AWS ec2 instance<\/strong>, make sure you add an IAM role with permissions<\/strong> to interact with AWS ec2 services. I have added the required IAM policy JSON in Github. See Packer IAM policy JSON<\/a><\/p>\n
          Step 5:<\/strong> Now let’s validate the Packer HCL template before we build it. Validation is very important in the CI process.<\/p>\n
          packer validate vm.pkr.hcl<\/code><\/pre>\n
          Step 6: <\/strong>Run the packer build using the following command.<\/p>\n
          packer build vm.pkr.hcl<\/code><\/pre>\n
          The above command runs for almost 3 – 5 minutes.<\/p>\n
          If you check your AMI list, you should see an AMI named PACKER-DEMO-httpd<\/code>.<\/p>\n
          Debugging Packer Build<\/h2>\n
          There are situations where you might need to debug a packer build.<\/p>\n
          The -debug flag lets you debug the AMI build.<\/p>\n
          For example,<\/p>\n
          packer build -debug vm.pkr.hcl<\/code><\/pre>\n
          When you run packer in debug mode, You need to approve each packer step manually as shown in the image below.<\/p>\n
          \"Packer<\/figure>\n
          Also, You can ssh into the ec2 instance<\/strong> created by Packer using the temporary ssh key or the ssh key pair provided in the packer config.<\/p>\n
          If you don’t specify the ssh key in the packer configuration, it creates a temporary ssh key and places it in the same folder where you run the packer command.<\/p>\n
          So that you can use the IP address and temporary ssh key to log in to the packer VM to debug the issues.<\/p>\n
          If you want to see how packer works in real world setup, you can checkout the following DevOps project<\/a> video where I have used Packer with Terraform.<\/p>\n

          \n