Skip to content

Azure Blob Storage Destination

Configure Azure Blob Storage as a destination for your data transfers in Pontoon.

Prerequisites

Before configuring Azure Blob Storage as a destination, ensure you have:

  • Azure Account
  • Azure Storage Account: An Azure Storage account with Hierarchical Namespace enabled
  • Storage Container: Go to Storage account > Data storage > Containers and create a new container if needed
  • Shared Access Signature (SAS): Connection String URI for Azure Storage with permission to read and write Blobs and Files in your container (Storage account > Security + networking > Shared access signature)

How it works

The Azure Blob Storage connector writes data to your bucket as compressed Apache Parquet files using Apache Hive style partitions, which is supported by most query engines and data platforms making it easy to work with.

This connector is append-only, so re-running syncs will produce new files with a later timestamp and different batch ID (see below) but will not delete existing data in the destination.

Structure of landed data

Data written to Azure will have the following structure:

abfss://<blob_container>/<blob_prefix>/<model>/dt=<transfer_date>/<transfer_timestamp>_<batch_id>_<file_index>.parquet

  • <blob_container> is the name of your Azure Storage Container
  • <blob_prefix> is an optional folder prefix
  • <model> is the name of the data model transferred, similar to a table name
  • <transfer_date> is the date that the transfer started in the format 2025-01-01
  • <transfer_timestamp> is a timestand of when the transfer started in the format 20250301121507
  • <batch_id> is a batch ID generated by Pontoon that is unique to the running transfer - subsequent transfers to the same destination will have different batch IDs
  • <file_index> is a monotonically increasing integer for a given batch ID

Configuration

Parameter Description Required Example
blob_container Azure Storage Container Name Yes my-container
blob_prefix Folder path prefix Optional /exports
blob_connection_string Azure SAS Connectio URI Yes BlobEndpoint=https://<account>.blob.core.windows.net/

Setup Destination

  1. Navigate to Destinations → New Destination
  2. Select S3 as the destination type
  3. Enter connection details:
  4. Click Test Connection to verify
  5. Click Save to create the destination