Skip to content

S3 Destination

Configure Amazon S3 as a destination for your data transfers in Pontoon.

Prerequisites

Before configuring S3 as a destination, ensure you have:

  • AWS Account
  • S3 Bucket: S3 bucket that data will be sent to
  • Credentials: AWS Credentials with s3:PutObject, s3:DeleteObject permissions for your bucket

How it works

The S3 connector writes data to your bucket as compressed Apache Parquet files using Apache Hive style partitions, which is supported by most query engines and data platforms making it easy to work with.

This connector is append-only, so re-running syncs will produce new files with a later timestamp and different batch ID (see below) but will not delete existing data in the destination.

Structure of landed data

Data written to S3 will have the following structure:

s3://<bucket_name>/<prefix>/<model>/dt=<transfer_date>/<transfer_timestamp>_<batch_id>_<file_index>.parquet

  • <bucket_name> is the name of your S3 bucket
  • <prefix> is an optional folder prefix
  • <model> is the name of the data model transferred, similar to a table name
  • <transfer_date> is the date that the transfer started in the format 2025-01-01
  • <transfer_timestamp> is a timestand of when the transfer started in the format 20250301121507
  • <batch_id> is a batch ID generated by Pontoon that is unique to the running transfer - subsequent transfers to the same destination will have different batch IDs
  • <file_index> is a monotonically increasing integer for a given batch ID

Configuration

Parameter Description Required Example
s3_bucket S3 Bucket Name Yes my-bucket
s3_prefix Folder path prefix Optional /exports
s3_region Region of bucket Yes us-east-1
aws_access_key_id AWS access key ID Yes AKIAIOSFODNN7EXAMPLE
aws_secret_access_key AWS secret access key Yes wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Setup Destination

  1. Navigate to Destinations → New Destination
  2. Select S3 as the destination type
  3. Enter connection details:
  4. Click Test Connection to verify
  5. Click Save to create the destination