Home

Automate S3 to EBS with PS

by Kolby

Posted on February 17, 2014

Tags: PowershellScriptsEBSS3Backups


Currently we have a few clients that utilize AWS as part of their DR solution (upcoming articles). Part of this process has us store the larger base images on EBS stores. This allows us to save time during the restore process. The code below will take a specific set of Keys (S3 term for Folders) and copy them to a local path, matching the folder name. This script requires that the AWS Powershell tools be installed.

# +---------------------------------------------------------------------------
# | File : CopyfromAWStoLocal.ps1
# | Version : 1.0
# | Purpose : Copy files from S3 to Local Storage
# | Synopsis:
# | Usage : .\CopyfromAWStoLocal.ps1
# +----------------------------------------------------------------------------
# |
# | File Requirements:
# | Must have AWS S3 CLI installed & Powershell tools
# | CLI - https://s3.amazonaws.com/aws-cli/AWSCLI64.msi
# | PS Tools - http://aws.amazon.com/powershell/
# +----------------------------------------------------------------------------
# | Maintenance History
# | View GitHub notes: https://github.com/allenk1/ISO-Scripts/commits/master/BackuptoS3_Snapshots.ps1
# ********************************************************************************


# Default input params
$access = "ABC12312312312312312"
$private = "ABCDEFGHIJ1231231231ABCDEFGHIJ1231231231"
$foldernames = @("Dir1", "Dir2", "Dir3")
$bucket = "bucketname"
$downloadpath = "E:" #no trailing slash
$region = "us-west-2"

import-module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"

# Clear any saved credentials
# Clear-AWSCredentials -StoredCredentials

# Set credentials
Set-AWSCredentials -AccessKey $access -SecretKey $private
Set-DefaultAWSRegion $region

foreach ($dir in $foldernames){

    $store = $downloadpath + '\' + $dir
    Read-S3object -BucketName $bucket -KeyPrefix $dir -Folder $store

}




Comments

Automating AWS Backups with PowerShell

by Kolby

Posted on February 16, 2014

Tags: PowershellScriptsEBSS3Backups


A few of my clients currenlty have some simple windows server on AWS. In order to save costs we wanted to utilize some of the storage tools that AWS provides. We settled on EBS snapshots for VM level backup and then doing a flat file copy to S3. For this particular client we enabled versioning on S3 in order to retain multiple version.

Here is the script to backup the files:

# +---------------------------------------------------------------------------
# | File : BackuptoS3_Snapshots.ps1
# | Version : 1.0
# | Purpose : Backs up to S3 & creates EBS snapshots
# | Synopsis:
# | Usage : .\BackuptoS3_Snapshots.ps1
# +----------------------------------------------------------------------------
# |
# | File Requirements:
# | Must have AWS S3 CLI installed & Powershell tools
# | CLI - https://s3.amazonaws.com/aws-cli/AWSCLI64.msi
# | PS Tools - http://aws.amazon.com/powershell/
# +----------------------------------------------------------------------------
# | Maintenance History
# | View GitHub notes: https://github.com/allenk1/ISO-Scripts/commits/master/BackuptoS3_Snapshots.ps1
# ********************************************************************************


# Default input params
$access = "AKIAJGXXXXXXXXXXXXXX"
$private = "ABC123123ABC123123ABC123123ABC123123ABCD"
$vol_id = @("vol-XXXXXXXX", "vol-XXXXXXXX")
$servername = "Server_NAME"
$region = "us-west-2"   # Regions: us-east-1, us-west-2, us-west-1, eu-west-1, ap-southeast-1
# ap-southeast-2, ap-northeast-1, sa-east-1

# $a = Get-Date
# $date = $a.Year + "_" + $a.Month + "_" + $a.Day   #YYYYMMDD
$date = Get-Date -format s

import-module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"

# Clear any saved credentials
# Clear-AWSCredentials -StoredCredentials

# Set credentials
Set-AWSCredentials -AccessKey $access -SecretKey $private
Set-DefaultAWSRegion $region

# Loop through all volumes and create snapshots
# Naming Scheme ServerName_VOLID

foreach ($vol in $vol_id) {

    # snapshot the EBS store
    $snapshot_name = $servername + "_" + $vol + "_" + $date

    New-EC2Snapshot -VolumeId $vol -Description $snapshot_name

}

# Now flat file copy to S3
# Enable Bucket versioning in order to keep mulitple version of the file
# TODO: Version with Script

$copy_dirs = @('C:\Path\to\Backup\Directory')
$bucket = "bucketname"

foreach ($dir in $copy_dirs){

    # Key setup by ServerName_DATE
    $key = $servername + '_' + $date
    Write-S3Object -Folder $dir -BucketName $bucket -KeyPrefix / -Recurse

}

Please make sure to set the following Variables:

  • $access - Your Access Key (from IAM)
  • $private - Your Private Key from (IAM)
  • $vol_id = @(“vol-XXXXXXXX”, “vol-XXXXXXXX”) - Array of EBS volumes to backup
  • $servername - Server Name for Backup labeling
  • $region - Decide what Region you’d like to backup to
  • $copy_dirs = @(‘C:\Path\to\Backup\Directory’) - Array of local path’s you’d like to setup
  • $bucket - Backup bucket name (make sure the creds above have full access)



Comments

Social