[FEATURE]: DBR upgrade mechanism

[chase-edwards-db]

There is no systematic way to upgrade DBR. Often, the first step large customers need to take is a DBR upgrade to move to a UC-supported runtime. Note that this feature could (and should) be built for the generic case, independent of UC migrations as historically it has always been a field need.

Related issues:

• [EPIC] Lint problems related to previous Databricks Runtime versions #1124
• [FEATURE]: Regression testing harness for post-migration validation of configurations (Workload Assets) #1122

Proposed Solution

The feature should:

• Evaluate and display unsupported DBR impact preventing UC migration
• Migrate clusters and associated policies to new target runtime
• Validate successful migration and associated workloads.

This can/should be rolled into #1122, where validation should be done via:

• Successful change of targeted cluster configuration.
• Successful execution of targeted workflows.

and should surface the following information:

• Errors upon failure of DBR upgrade
• Migration rollup success metrics (% clusters migrated successfully, % migrated jobs executed successfully, % migrated workflows executed successfully)
• Tables with item-level success metrics e.g. for clusters (cluster_id | migrated_flag | error_message) and for workflows (job_id, execution_success, error_message)

Additional Context

For one enterprise customer, it took nearly 12 months to move from 7.3LTS to 9.1LTS.

[nfx]

@chase-edwards-db how can validation be automated? Please describe

[chase-edwards-db]

Updated, let me know if more info needed @nfx

[pohlposition]

This seems like a HUGE feature request. Like it could be it's own Labs project.
I'm not suggesting it's not important, just a large amount of work; and somewhat independent to the goals of UCX.