Task: Setup Google Cloud VM to run the revised pipeline

[cfreedman]

Need to look into the GC account to either restructure the existing resource there or create a new VM that's scheduled to run the new pipeline on a monthly basis. We still need to make decision regarding how this will be setup i.e. simple cron job or some other infrastructure.

[nlebovits]

Generally, based on previous conversations, this will involve:

VM to Cloud Run Job Migration Instructions

• Artifact Registry: Create Docker repository to store container images built from GitHub
• Service Account: Create GCP service account with Cloud Run + Storage + Artifact Registry permissions, download JSON key
• GitHub Secrets: Add service account key, project ID, and other pipeline secrets to repository
• GitHub Actions: Set up monthly scheduled workflow that builds container, pushes to Artifact Registry, and executes Cloud Run Job
• Cloud Run Job: Create job with resource settings based on your local pipeline requirements (CPU/memory), set timeout to match pipeline duration, configure environment variables
• Code Updates: Update Slack notifications for Cloud Run context, ensure data outputs to correct GCS paths
• Testing: Test manual GitHub Actions trigger, verify Cloud Run Job execution, confirm data output and Slack notifications work
• Documentation: Update deployment docs to reflect new Cloud Run architecture
• Cleanup: Disable old VM cron job, stop VM instance, remove old backup scripts