Skip to content

fix: use retryable http client #5315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 8, 2025

Conversation

emosbaugh
Copy link
Member

@emosbaugh emosbaugh commented May 8, 2025

What this PR does / why we need it:

APIs can be flakey. Use retryable http client for resilliency.

https://github.com/replicatedhq/embedded-cluster/actions/runs/14914714913/job/41898012325

{"level":"error","ts":"2025-05-08T19:18:55Z","msg":"failed to check if upgrade service can start: failed to get available updates: failed to get updates: failed to list replicated app releases: <html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n"}
{"level":"info","ts":"2025-05-08T19:18:55Z","msg":"method=POST status=500 duration=5.687813709s request=/api/v1/app/embedded-cluster-smoke-test-staging-app/start-upgrade-service"}

Which issue(s) this PR fixes:

Does this PR require a test?

Does this PR require a release note?


Does this PR require documentation?

@emosbaugh emosbaugh added type::bug Something isn't working bug::normal labels May 8, 2025
@emosbaugh emosbaugh marked this pull request as ready for review May 8, 2025 22:06
if err != nil {
return "", errors.Wrap(err, "failed to create new request")
}

reportingInfo := reporting.GetReportingInfo(appID)
reporting.InjectReportingInfoHeaders(req, reportingInfo)
reporting.InjectReportingInfoHeaders(req.Header, reportingInfo)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this set the reporting headers correctly on the request because Header is a map and maps are passed by reference? don't wanna suddenly lose reporting information

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it in a go playground

https://go.dev/play/p/kyXbg_CvyNH

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a unit test

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

pkg/util/http.go Outdated
if bodyStr != "" {
return resp, fmt.Errorf("%s %s giving up after %d attempt(s): %s",
req.Method, redactURL(req.URL), attempt, bodyStr)
} else {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could get rid of this else keyword for clarity and return directly

pkg/util/http.go Outdated
if bodyStr != "" {
return resp, fmt.Errorf("%s %s giving up after %d attempt(s) with error %w: %s",
req.Method, redactURL(req.URL), attempt, err, bodyStr)
} else {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

pkg/util/http.go Outdated
header.Add("User-Agent", buildversion.GetUserAgent())
}

// errorHandler includes the body in the response when there is an error
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would this cause OOM if the body was a non-complete binary download? should probably have a cap on the body size that can be read

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call. i made this much more simple

sgalsaleh
sgalsaleh previously approved these changes May 8, 2025
@emosbaugh emosbaugh merged commit 1828357 into main May 8, 2025
92 checks passed
@emosbaugh emosbaugh deleted the emosbaugh/20250508/retryable-http-client branch May 8, 2025 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug::normal type::bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants