Skip to content

Postgres and FORCE_RELOAD issues in the new pipeline #1152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamzev opened this issue Mar 30, 2025 · 0 comments
Open

Postgres and FORCE_RELOAD issues in the new pipeline #1152

adamzev opened this issue Mar 30, 2025 · 0 comments

Comments

@adamzev
Copy link
Collaborator

adamzev commented Mar 30, 2025

There are bugs relating to loading data to and from Postgres in the new ETL pipeline.

  • ALTER statements occur prior to table creation when FORCE_RELOAD is True
  • data is appended to the tables which store the API responses rather than being replaced, however when querying for data, all data is fetched (so reloading from the database gets the API responses of every run rather than the most recent)
  • In pwd_parcels there is a subtle bug where we return FeatureLayer(name=primary_featurelayer.name, gdf=merged_gdf). If FORCE_RELOAD is False, this triggers a fetch of the table opa_properties so rather than setting gdf to merged_gdf, it is set to the API request results from a couple steps back. There is an easy patch to this but we may want to look into this in more detail to prevent unexpected behavior or address this in Task: Add subclasses to FeatureLayer #1143

If needed, I'll add more details to this ticket or break down the issues into separate tickets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: To Do
Development

No branches or pull requests

1 participant