Connect Google Cloud Storage

Connect Horizon Epoch to Google Cloud Storage for Delta Lake tables.

CLI Configuration

# Add GCS storage backend
epoch storage add datalake \
    --type gcs \
    --bucket "my-gcs-bucket" \
    --project "my-project-id"

Authentication Methods

Service Account Key

# Set credentials file path
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

epoch storage add datalake \
    --type gcs \
    --bucket "my-gcs-bucket" \
    --project "my-project-id"

Or specify directly:

epoch storage add datalake \
    --type gcs \
    --bucket "my-gcs-bucket" \
    --project "my-project-id" \
    --credentials-file "/path/to/service-account.json"

Application Default Credentials (ADC)

For GCE, GKE, Cloud Run, or local development with gcloud auth:

# Local development: authenticate with gcloud
gcloud auth application-default login

epoch storage add datalake \
    --type gcs \
    --bucket "my-gcs-bucket" \
    --project "my-project-id" \
    --auth-method adc

Workload Identity (GKE)

For GKE with Workload Identity configured:

epoch storage add datalake \
    --type gcs \
    --bucket "my-gcs-bucket" \
    --project "my-project-id" \
    --auth-method adc

Register Delta Tables

epoch table add events \
    --location "gcs://datalake/delta/events"

Table Storage Structure

Tables are stored as Delta Lake format:

bucket/
  delta/
    events/
      _delta_log/
      part-00000.parquet
      part-00001.parquet

Configuration File

# ~/.epoch/config.toml
[storage.gcs.datalake]
bucket = "my-gcs-bucket"
project_id = "my-project-id"
prefix = "horizon-epoch/"
auth_method = "service_account"
credentials_file = "/path/to/service-account.json"

Required Permissions

Minimum IAM permissions for the service account:

roles/storage.objectViewer  # Read operations
roles/storage.objectCreator # Write operations
roles/storage.objectAdmin   # Full access (recommended)

Or custom role with:

• storage.objects.get
• storage.objects.list
• storage.objects.create
• storage.objects.delete

Troubleshooting

Permission Denied

• Verify service account has required IAM roles
• Check bucket-level permissions
• Ensure project ID is correct

Bucket Not Found

• Verify bucket name is correct
• Check bucket is in the correct project

Credentials Not Found

• Set GOOGLE_APPLICATION_CREDENTIALS environment variable
• Or use --credentials-file parameter
• For ADC, run gcloud auth application-default login