Skip to content

Task: Setup Google Cloud VM to run the revised pipeline #1201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cfreedman opened this issue May 14, 2025 · 1 comment
Open

Task: Setup Google Cloud VM to run the revised pipeline #1201

cfreedman opened this issue May 14, 2025 · 1 comment

Comments

@cfreedman
Copy link
Collaborator

Need to look into the GC account to either restructure the existing resource there or create a new VM that's scheduled to run the new pipeline on a monthly basis. We still need to make decision regarding how this will be setup i.e. simple cron job or some other infrastructure.

@nlebovits
Copy link
Collaborator

Generally, based on previous conversations, this will involve:

VM to Cloud Run Job Migration Instructions

  • Artifact Registry: Create Docker repository to store container images built from GitHub
  • Service Account: Create GCP service account with Cloud Run + Storage + Artifact Registry permissions, download JSON key
  • GitHub Secrets: Add service account key, project ID, and other pipeline secrets to repository
  • GitHub Actions: Set up monthly scheduled workflow that builds container, pushes to Artifact Registry, and executes Cloud Run Job
  • Cloud Run Job: Create job with resource settings based on your local pipeline requirements (CPU/memory), set timeout to match pipeline duration, configure environment variables
  • Code Updates: Update Slack notifications for Cloud Run context, ensure data outputs to correct GCS paths
  • Testing: Test manual GitHub Actions trigger, verify Cloud Run Job execution, confirm data output and Slack notifications work
  • Documentation: Update deployment docs to reflect new Cloud Run architecture
  • Cleanup: Disable old VM cron job, stop VM instance, remove old backup scripts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: To Do
Development

No branches or pull requests

2 participants