v0.59.0

• Adds requirement for matching account groups to be created before assessment to the docs (#4017). The account group setup requirements have been clarified to ensure successful assessment and group migration workflows, mandating that account groups matching workspace local groups are created beforehand, which can be achieved manually or programmatically via various methods. The assessment workflow has been enhanced to retrieve workspace assets and securable objects from the Hive metastore for compatibility assessment with UC, storing the results in the inventory database for further analysis. Additionally, the documentation now stresses the necessity of running the validate-groups-membership command prior to initiating the group migration workflow, and recommends running the create-account-groups command beforehand if the required account groups do not already exist, to guarantee a seamless execution of the assessment and migration processes.
• Fixed Service Principal instructions for installation (#3967). The installation requirements for UCX have been updated to reflect changes in Service Principal support, where it is no longer supported for workspace installations, but may be supported for account-level installations. As a result, account-level identity setup now requires connection via Service Principal with Account Admin and Workspace Admin privileges in all workspaces. All other installation requirements remain unchanged, including the need for a Databricks Premium or Enterprise workspace, network access to the Databricks Workspace and the Internet, a created Unity Catalog Metastore, and a PRO or Serverless SQL Warehouse for rendering reports. Additionally, users with external Hive Metastores, such as AWS Glue, must consult the relevant guide for specific instructions to ensure proper setup.
• Fixed migrate tables when default catalog is set (#4012). The handling of the default catalog in the Hive metastore has been enhanced to ensure correct behavior when the default catalog is set. Specifically, the DESCRIBE SCHEMA EXTENDED and SHOW TBLPROPERTIES queries have been updated to include the hive_metastore prefix when fetching database descriptions and constructing table identifiers, respectively, unless the table is located in a mount point, in which case the delta prefix is used. This change addresses a previously reported issue with migrating tables when the default catalog is set, ensuring that table properties are correctly fetched and tables are properly identified. The update has been applied to multiple test cases, including those for skipping tables, upgraded tables, and mapping tables, to guarantee correct execution of queries with the default catalog name, which is essential when the default catalog is set to hive_metastore.
• Limit crawl workflows task in assessment to workflows that ran in the last 30 days (#3963). The JobInfo class has been enhanced with a new last_run attribute to store the timestamp of the last pipeline execution, allowing for better monitoring and assessment. The from_job method has been updated to initialize this attribute consistently. Additionally, the assess_workflows method now filters workflows to only include those that have run within the last 30 days, achieved through the introduction of a last_run_days parameter in the refresh_report method. This parameter enables time-based filtering of job runs, and a new inner function lint_job_limited handles the filtering logic. The lint_job method has also been updated to accept the last_run_days parameter and check if a job has run within the specified time frame. Furthermore, a new test method test_workflow_linter_refresh_report_time_bound has been added to verify the correct functioning of the WorkflowLinter class when limited to recent workflow runs, ensuring that it produces the expected results and writes to the correct tables.
• Pause migration progress workflow schedule (#3995). The migration progress workflow schedule is now paused by default, with its pause_status set to PAUSED, to prevent automatic execution and potential failures due to missing prerequisites. This change is driven by the experimental nature of the workflow, which may fail if a UCX catalog has not been created by the customer. To ensure successful execution, users are advised to unpause the workflow after running the create-ucx-catalog command, allowing them to control when the workflow runs and verify that necessary prerequisites are in place.
• Warns instead of an error while finding an acc group in workspace (#4016). The behavior of the account group reflection functionality has been updated to handle duplicate groups more robustly. When encountering a group that already exists in the workspace, the function now logs a warning instead of an error, allowing it to continue executing uninterrupted. This change accommodates the introduction of nested account groups from workspace local groups, which can lead to groups being present in the workspace that are also being migrated. The warning message clearly indicates that the group is being skipped due to its existing presence in the workspace, providing transparency into the reflection process.

Contributors: @pritishpai, @FastLee