{"id":1053,"date":"2021-06-15T02:10:00","date_gmt":"2021-06-15T02:10:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=1053"},"modified":"2021-06-15T02:10:00","modified_gmt":"2021-06-15T02:10:00","slug":"automate-ebs-snapshot-creation-deletion","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=1053","title":{"rendered":"How to Automate EBS Snapshot Creation [ AWS Ec2 Backup ]"},"content":{"rendered":"

It is essential to have AWS ec2 backup using EBS volumes for data recovery and protection. AWS EBS snapshots play an important role in the backup of your ec2 instance data (root volumes & additional volumes).<\/p>\n

Even though AWS EBS snapshots are considered “poor man’s backup,” it gives you a point in time backup and faster restore options to meet your RPO<\/a> objective.<\/p>\n

Towards the end of the article, I have added some key EBS snapshot features and some best practices to manage snapshots.<\/p>\n

Automation AWS EBS Snapshot For EC2 Instances<\/h2>\n

AWS EBS Snapshots are the cheapest and easiest way to enable backups for your EC2 instances or EBS volumes.<\/p>\n

There are three ways to take automated snapshots.<\/p>\n

    \n
  1. AWS EBS Life Cycle manager<\/li>\n
  2. AWS Cloudwatch Events<\/li>\n
  3. AWS Lambda Functions.<\/li>\n<\/ol>\n

    This tutorial will guide you through EBS snapshot automation and its life cycle management using all three approaches.<\/p>\n

    Automation EBS Snapshot \u00a0with Life Cycle manager<\/h2>\n

    AWS EC2 lifecycle manage is a native AWS functionality to manage the lifecycle of EBS volumes and snapshots.<\/p>\n

    It is the quickest and easiest way to automate EBS snapshots. It works on the concept of AWS tags. Based on the instance or volume tags, you can group EBS volumes and perform snapshot operations in bulk or for a single instance.<\/p>\n

    Follow the steps given below to setup a EBS snapshot lifecycle policy.<\/p>\n

    Step 1: Tag your ec2 instance and \u00a0volumes<\/strong><\/h3>\n

    EC2 EBS snapshots with the life cycle manager work with the instance & volume tags. It requires instances and volumes to be tagged to identify the snapshot candidate.<\/p>\n

    You can use the following tag in the instances and volumes that needs automated snapshots.<\/p>\n

    Key = Backup \nValue = True<\/code><\/pre>\n
    \"AWS<\/figure>\n
    Step 2: Go to the EBS life cycle manager to create a snapshot lifecycle policy.<\/h3>\n
    Head over to the EC2 dashboard and select “Lifecycle Manager<\/code>” option under ELASTIC BLOCK STORE<\/code> category as shown below.<\/p>\n
    \"EBS<\/figure>\n
    You will be taken to the life cycle manager dashboard. Click \u00a0“Create Snapshot Lifecycle Policy<\/code>” button.<\/p>\n
    \"life<\/figure>\n
    Step 3: Add EBS snapshot life cycle policy rules<\/h3>\n
    Enter the EBS snapshot policy details as shown below. Ensure that you select the right tags for the volumes you need the snapshot.<\/p>\n
    Note: <\/strong>You can add multiple tags to target specific Volumes<\/p><\/blockquote>\n
    \"Life<\/figure>\n
    Enter EBS snapshot schedule details based on your requirements. You can choose retention type for both count & age.<\/p>\n
    For regular ec2 ebs backups, count is the ideal way.<\/p>\n
    Also apply proper tags to identify the snapshots.<\/p>\n
    \"ec2<\/figure>\n
    There are two optional parameters for the EBS snapshot high availability and fast restore. You can choose these options for production volumes.<\/p>\n
    Keep in mind that these two options will incur extra charges.<\/p>\n
    \"High<\/figure>\n
    Select an IAM role that has permission to create and delete snapshots. If you don’t have an IAM role, you can use the default role option. AWS will automatically create a role for snapshots.<\/p>\n
    I recommend you to create a custom role and use it with the policy to keep track of IAM roles.<\/p>\n
    Also select “enable policy” for the policy to be active immediately after creation.<\/p>\n
    <\/figure>\n
    Click create policy.<\/p>\n
    Now the policy manager will automatically create snapshots based on the schedules you have added.<\/p>\n
    Create EBS Volume Snapshots With Cloudwatch Events<\/h2>\n
    Cloudwatch custom events & \u00a0schedules can be used to create EBS snapshots.<\/p>\n
    You can choose AWS services events for cloudwatch to trigger custom actions.<\/p>\n
    To demonstrate this, I will use the cloudwatch schedule to create EBS snapshots. Follow the steps given below.<\/p>\n
    [powerkit_posts title=”Also Read” \u00a0ids=”6350″ \u00a0image_size=”pk-thumbnail” template=”list”]<\/p>\n
    First we need to create a Cloudwatch<\/a> Schedule.<\/p>\n
    Head over to cloudwatch service and click create a rule under the rule options as shown below.<\/p>\n
    \"cloudwatch<\/figure>\n
    You can choose either a fixed schedule or a cron expression. Under targets, search for ec2 and select the “EC2 CreateSnapshot API Call” option.<\/p>\n
    Get the Volume ID from the EBS volume information, apply it to the Volume ID field and click “Configure details”.<\/p>\n
    Create more targets if you want to take snapshot of more volumes.<\/p>\n
    \"cloudwatch<\/figure>\n
    Enter the rule name, description and click create rule.<\/p>\n
    \"cloudwatch<\/figure>\n
    Thats it. Based on the cloudwatch schedules, the snapshots will be created.<\/p>\n
    Automate EBS snapshot Creation and Deletion With Lambda Function<\/h2>\n
    If you have any use case where the lifecycle manager does not suffice the requirements, you can opt for lambda-based snapshot creation. Most use cases come under unscheduled activities.<\/p>\n
    One use case I can think of is, taking snapshots just before updating\/upgrading stateful systems. You can have an automation that will trigger a lambda function that performs the snapshot action.<\/p>\n
    Getting Started With Lambda Based EBS snapshot<\/h3>\n
    We will use Python 2.7 scripts, lambda, IAM role, and cloud watch event schedule for this setup.<\/p>\n
    For this lambda function to work, you need to create a tag named “backup” with the value true for all the instances for which you need a backup.<\/p>\n
    For setting up a lambda function for creating automated snapshots, you need to do the following.<\/p>\n
      \n
    1. A snapshot creation python script with the necessary parameters.<\/li>\n
    2. An IAM role with snapshot create, modify, and delete access.<\/li>\n
    3. A lambda function.<\/li>\n<\/ol>\n
      Configure Python Script for EBS Snapshot<\/h3>\n
      Following python code will create snapshots on all the instance which have a tag named “backup.”<\/p>\n
      Note: <\/strong>You can get all the code from here<\/a><\/em><\/p><\/blockquote>\n
      import boto3\nimport collections\nimport datetime\n\nec = boto3.client('ec2')\n\ndef lambda_handler(event, context):\n    reservations = ec.describe_instances(\n        Filters=[\n            {'Name': 'tag-key', 'Values': ['backup', 'Backup']},\n        ]\n    ).get(\n        'Reservations', []\n    )\n\n    instances = sum(\n        [\n            [i for i in r['Instances']]\n            for r in reservations\n        ], [])\n\n    print \"Found %d instances that need backing up\" % len(instances)\n\n    to_tag = collections.defaultdict(list)\n\n    for instance in instances:\n        try:\n            retention_days = [\n                int(t.get('Value')) for t in instance['Tags']\n                if t['Key'] == 'Retention'][0]\n        except IndexError:\n            retention_days = 10\n\n        for dev in instance['BlockDeviceMappings']:\n            if dev.get('Ebs', None) is None:\n                continue\n            vol_id = dev['Ebs']['VolumeId']\n            print \"Found EBS volume %s on instance %s\" % (\n                vol_id, instance['InstanceId'])\n\n            snap = ec.create_snapshot(\n                VolumeId=vol_id,\n            )\n\n            to_tag[retention_days].append(snap['SnapshotId'])\n\n            print \"Retaining snapshot %s of volume %s from instance %s for %d days\" % (\n                snap['SnapshotId'],\n                vol_id,\n                instance['InstanceId'],\n                retention_days,\n            )\n\n\n    for retention_days in to_tag.keys():\n        delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)\n        delete_fmt = delete_date.strftime('%Y-%m-%d')\n        print \"Will delete %d snapshots on %s\" % (len(to_tag[retention_days]), delete_fmt)\n        ec.create_tags(\n            Resources=to_tag[retention_days],\n            Tags=[\n                {'Key': 'DeleteOn', 'Value': delete_fmt},\n                {'Key': 'Name', 'Value': \"LIVE-BACKUP\"}\n            ]\n        )\n\ufeff<\/code><\/pre>\n
      Also, you can decide on the retention time for the snapshot.<\/p>\n
      By default, the code sets the retention days as 10. If you want to reduce or increase the retention time, you can change the following parameter in the code.<\/p>\n
      retention_days = 10<\/p>\n
      The python script will create a snapshot with a tag key “Deletion” and “Date” as the value that is calculated based on the retention days. This will help in deleting the snapshots which are older than the retention time.<\/p>\n
      Lambda Function To Automate Snapshot Creation<\/h3>\n
      Now that we have our python script ready for creating snapshots, it has to deployed as a Lambda function.<\/p>\n
      Triggering the Lambda function totally depends on your use case.<\/p>\n
      For demo purposes, we will set up cloudwatch triggers to execute the lambda function whenever a snapshot is required.<\/p>\n
      Follow the steps given below for creating a lambda function.<\/p>\n
      Step 1: <\/strong>Head over to lambda service page and select “create lambda function”.<\/p>\n
      \"lambda<\/figure>\n
      Step 2: <\/strong>Choose “Author from Scratch” and python 2.7 runtime. Also, select an exiting IAM role with snapshot create permissions.<\/p>\n
      Click “Create Function” function button after filling up the details.<\/p>\n
      \"create<\/figure>\n
      Step 3: <\/strong>On the next page, if you scroll down, you will find the function code editor. Copy the python script from the above section to the editor and save it.<\/p>\n
      \"Save<\/figure>\n
      Once saved, click the “Test” button. It will open an evet pop up. Just enter an event name and click create it.<\/p>\n
      \"Create<\/figure>\n
      Click “Test” button again and you will see the code getting executed and its logs as show blow. As per the code, it should create snapshots of all volumes if a instance has a tag named “Backup:True”.<\/p>\n
      \"Test<\/figure>\n
      Step 4:<\/strong> Now you have a Lamda function ready to create snapshots.<\/p>\n
      You have to decide what triggers you need to invoke the lambda function. If you click the “Add Trigger” Button from the function dashboard, it will list all the possible trigger options as shown below. You can configure one based on your use case. It can be API gateway wall or a cloudwatch even trigger like I explained above.<\/p>\n
      \"Add<\/figure>\n
      For example, I if choose cloudwatch event trigger, It will look like the following.<\/p>\n
      \"Add<\/figure>\n
      EBS Snapshot Deletion Automation Using Lambda<\/h3>\n
      We have seen how to create a lambda function and create \u00a0EBS snapshots of ec2 instances tagged with a “backup<\/code>” tag. We cannot keep the snapshots piling up over time. That’s the reason we used the retention days in the python code. It tags the snapshot with the deletion date.<\/p>\n
      The deletion python script scans for snapshots with a tag with a value that matches the current date. If a snapshot matches the requirement, it will delete that snapshot. This lambda function runs every day to remove the old snapshots.<\/p>\n
      Create a lambda function with the cloudwatch event schedule as one day. You can follow the same steps I explained above for creating the lambda function.<\/p>\n
      Here is the python code for snapshot deletion.<\/p>\n
      import boto3\nimport re\nimport datetime\n\nec = boto3.client('ec2')\niam = boto3.client('iam')\n\ndef lambda_handler(event, context):\n    account_ids = list()\n    try:\n        \"\"\"\n        You can replace this try\/except by filling in `account_ids` yourself.\n        Get your account ID with:\n        > import boto3\n        > iam = boto3.client('iam')\n        > print iam.get_user()['User']['Arn'].split(':')[4]\n        \"\"\"\n        iam.get_user()\n    except Exception as e:\n        # use the exception message to get the account ID the function executes under\n        account_ids.append(re.search(r'(arn:aws:sts::)([0-9]+)', str(e)).groups()[1])\n\n    delete_on = datetime.date.today().strftime('%Y-%m-%d')\n    filters = [\n        {'Name': 'tag-key', 'Values': ['DeleteOn']},\n        {'Name': 'tag-value', 'Values': [delete_on]},\n    ]\n    snapshot_response = ec.describe_snapshots(OwnerIds=account_ids, Filters=filters)\n\n    for snap in snapshot_response['Snapshots']:\n        print \"Deleting snapshot %s\" % snap['SnapshotId']\n        ec.delete_snapshot(SnapshotId=snap['SnapshotId'])<\/code><\/pre>\n
      How To Restore EBS Snapshot<\/h2>\n
      You can restore a snapshot in two ways.<\/p>\n
        \n
      1. Restore the EBS Volume from the snapshot.<\/li>\n
      2. Restore EC2 Instance from a snapshot<\/li>\n<\/ol>\n
        You can optionally change \u00a0following while restoring a snapshot<\/p>\n
          \n
        1. Volume Size<\/li>\n
        2. Disk Type<\/li>\n
        3. Availability Zone<\/li>\n<\/ol>\n
          Restore EBS Volume from a EBS Snapshot<\/h2>\n
          Follow the steps given below to restore a snapshot to a EBS volume.<\/p>\n
          Step 1:<\/strong> Head over to snapshots, select the snapshot you want to restore, select the “Actions” dropdown, and click create volume.<\/p>\n
          \"create<\/figure>\n
          Step 2: <\/strong>Fill in the required details and click “create volume” option.<\/p>\n
          \"EBS<\/figure>\n
          That’s it. Your volume will be created. You can mount this volume<\/a> to the required instance to access its data.<\/p>\n
          Restore EC2 Instance From EBS Snapshot<\/h2>\n
          You can restore an ec2 instance from a EBS snapshot with two simple steps.<\/p>\n
            \n
          1. Create an AMI <\/a>(ec2 machine Image) from the snapshot.<\/li>\n
          2. Launch an instance from the AMI created from the snapshot.<\/li>\n<\/ol>\n
            Follow the below steps.<\/p>\n
            Step 1:<\/strong> Head over to snapshots, select the snapshot you want to restore, select the “Actions<\/code>” dropdown, and click create an image.<\/p>\n
            \"Create<\/figure>\n
            Step 2: <\/strong>Enter the AMI name, description, and modify the required parameters. Click “Create Image<\/code>” to register the AMI.<\/p>\n
            \"parameters<\/figure>\n
            Step 3:<\/strong> Now, select AMIs from the left panel menu, select the AMI, and from the “Actions” drop-down, select launch.<\/p>\n
            It will take you to the generic instance launch wizard. You can launch the VM as you normally do with any ec2 instance creation.<\/p>\n
            \"Launch<\/figure>\n
            AWS EBS Snapshot Features<\/h2>\n
            Following are the key features of EBS snapshots.<\/p>\n
              \n
            1. Snapshots Backend Storage is s3:<\/strong> \u00a0Whenever you take a snapshot, it gets stored in S3.<\/li>\n
            2. EBS snapshots are incremental:<\/strong> Every time you request a Snapshot of your EBS volume, only the changed data in the disk (delta) is copied to the new one. So irrespective of the number of snapshots, you will only pay to changed data present in the Volume. Meaning, your consistent data never gets duplicated between Snapshots. For example, your disk storage can be 20 GB, and snapshot storage can be 30 GB due to the changes notified during every snapshot creation. You can read more about this here <\/a><\/li>\n<\/ol>\n
              \"AWS<\/figure>\n
              EBS Snapshot Best Practices<\/h2>\n
              Following are some best practices you can follow to manage EBS snapshots.<\/p>\n
                \n
              1. Standard Tagging: <\/strong>Tag your EBS volumes with standard tags across all your environments. This helps in a well-managed snapshot lifecycle management using the life cycle manager. Tags also help in tracking the cost<\/strong> associated with snapshots. You can have billing reposts based on tags.<\/li>\n
              2. Application Data Consistency:<\/strong> To have consistency for your snapshot backups, it is recommended to stop the IO activity on your disk and perform the disk snapshot.<\/li>\n
              3. Simultaneous Snapshot request:<\/strong> Snapshots do not affect disk performance, however, the simultaneous request could affect the disk performance.<\/li>\n<\/ol>\n
                Conclusion<\/h2>\n
                In this article we have seen three methods to post an EBS snapshot automation.<\/p>\n
                You can choose a method that suits your project needs, which complies with the organization’s security standards.<\/p>\n
                Either you are looking for a solution to take snapshots or trying to migrate to snapshot lifecycle manager.<\/p>\n
                Either way, leave a comment below!<\/p>\n
                \n
                Ngu\u1ed3n:<\/strong> How to Automate EBS Snapshot Creation  [ AWS Ec2 Backup ] \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
                Source: https:\/\/devopscube.com\/automate-ebs-snapshot-creation-deletion\/<\/p>\n","protected":false},"author":0,"featured_media":1054,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1053","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/1053","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1053"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/1053\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/1054"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1053"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1053"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1053"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}