There are a number of popular cloud providers, but migrating your data from one to another can be daunting.
At Kevel, by default our Data Shipping feature (which gives customers full logs of all ad serving requests) stores info in an AWS S3 bucket. But sometimes our customers request GCP (Google Cloud Platform) instead.
I started by weighing three data shipping options:
After weighing these options, I decided to proceed with an S3-triggered Lambda job, as it offered the best balance, the ability to handle scale at a reasonable price, and the ability to ‘upgrade’ if needed.
This step required a service account with limited permissions - similar to the IAM role for AWS. I had to embed JSON credentials in the Lambda package prior to upload.
Next, I weighed the deployment options - CloudFormation or Serverless.
CloudFormation is a simple, built-in option. It features native AWS tooling but requires pre-packaged deployment assets, and it doesn’t have native support for S3 events on existing buckets.
Serverless is a Cloud Agnostic Framework for developing and deploying cloud applications. It also includes handy tools for managing deployments including different environments. It’s easy to use with an existing S3 bucket, but it requires a third-party application to deploy it.
You, too, can you flip your data shipping from AWS to GCP by following these 10 steps.
Step One: Ensure that you have the Make and Zip command line tools installed.
On MacOS, you'll need to install the XCode Command Line Tools using:
xcode-select --install and Zip via Homebrew using:
brew update && brew install zip
If you're using an Ubuntu-based Linux distribution, you can run
sudo apt update && sudo apt install zip build-essential.
Step Two: Next you'll need the AWS CLI. You can find installation and configuration instructions here.
Once installed and configured, you can run
aws s3 mb s3://your-globablly-unique-lambda-source-bucket-name to create an S3 Bucket to store your Lambda source bundles
Step Three: Enable and configure data shipping for your network
Step Four: Now we need somewhere for the data to go. If you already have a GCP Storage bucket created, great, you can skip to creating the Service Account Key. If not, go ahead and create one in your account.
Step Five: Access or create your Service Account Key, which the Connector will use to write JSON files to your bucket (see note below)
Step Six: Save the key locally and set the Google Application Credentials environment variable to the path of the JSON file (You’ll need to remove any spaces in your JSON token filename for the make tasks)
NOTE: For Kevel customers, data shipping may re-write files using the same filename. To create new files in Google Cloud, grant the Storage Object Creator role when generating your token. You can allow the Connector to overwrite files by designating it Storage Object Admin.
Step One: Gather the following values for deployment:
Step Two: Start your deployment
Step Three: Create the stack:
make create STACK_NAME=your-stack-name \ LAMBDA_SOURCE_BUCKET=your-source-bucket \ SOURCE_BUCKET=your-adzerk-data-bucket \ DESTINATION_BUCKET=your-gcp-storage-bucket
Step Four: Update the stack:
make update STACK_NAME=your-stack-name \ LAMBDA_SOURCE_BUCKET=your-source-bucket \ SOURCE_BUCKET=your-adzerk-data-bucket \ DESTINATION_BUCKET=your-gcp-storage-bucket
The Google CloudFormation template in AWS will create new Lambda functions and infrastructure:
Once your new Google Cloud Storage bucket is installed, you can also set triggers to alert new data and execute further processing logic.
Questions or feedback about Kevel data shipping? Your Account Manager is always glad to hear from you!
Justin is a Senior Open Source Engineer at Kevel working on community building and better tools to enable customer success.