Code

Add MigrationSequencer for jobs#3008

Code

Merged

nfx merged 106 commits intomaindatabrickslabs/ucx:mainfrom

migration-sequencing-phase-1databrickslabs/ucx:migration-sequencing-phase-1Copy head branch name to clipboard

Nov 1, 2024

Commits

Commits on Nov 1, 2024

• make simple_dependency_resolver available more broadly

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  9539244

  Copy full SHA for 9539244

• build migration steps for workflow task

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  1740e53

  Copy full SHA for 1740e53

• fix pylint warnings

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  186ea87

  Copy full SHA for 186ea87

• fix pylint warnings

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  f01c4c0

  Copy full SHA for f01c4c0

• add object name

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  6613d6e

  Copy full SHA for 6613d6e

• populate object owner

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  0252ed4

  Copy full SHA for 0252ed4

• be more defensive

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  9f5705a

  Copy full SHA for 9f5705a

• move last_node_id to sequencer

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  1cee0ce

  Copy full SHA for 1cee0ce

• cherry-pick changes

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  069b4d9

  Copy full SHA for 069b4d9

• use existing Ownership classes

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  2472d35

  Copy full SHA for 2472d35

• fix merge issues

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  446fab5

  Copy full SHA for 446fab5

• move package

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  a19f939

  Copy full SHA for a19f939

• improve assert style

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  5bd3b67

  Copy full SHA for 5bd3b67

• formatting

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  1087463

  Copy full SHA for 1087463

• make 'incoming' transient and improve comments

  

  ericvergnaud

  authored and

  JCZuurmond

  committed
  77af278

  Copy full SHA for 77af278

• Lint unit test

  JCZuurmond

  committed
  8478d26

  Copy full SHA for 8478d26

• Test all steps in sequence

  JCZuurmond

  committed
  66e7122

  Copy full SHA for 66e7122

• Sort step ids

  JCZuurmond

  committed
  cfb555e

  Copy full SHA for cfb555e

• Add docstring for migration sequencing

  JCZuurmond

  committed
  c5a27f4

  Copy full SHA for c5a27f4

• Format

  JCZuurmond

  committed
  e7ff130

  Copy full SHA for e7ff130

• Update docstring

  JCZuurmond

  committed
  ad8f457

  Copy full SHA for ad8f457

• Use priority queue

  JCZuurmond

  committed
  00f4dc1

  Copy full SHA for 00f4dc1

• Add custom queue

  JCZuurmond

  committed
  08a676a

  Copy full SHA for 08a676a

• Add source

  JCZuurmond

  committed
  29555e3

  Copy full SHA for 29555e3

• Fix implementation of custom queue

  JCZuurmond

  committed
  7c9e770

  Copy full SHA for 7c9e770

• Update docs

  JCZuurmond

  committed
  e2be4a3

  Copy full SHA for e2be4a3

• Comment counter

  JCZuurmond

  committed
  652d173

  Copy full SHA for 652d173

• Add typehint for migration node id

  JCZuurmond

  committed
  d2505af

  Copy full SHA for d2505af

• Add docstrings to MigrationNodes

  JCZuurmond

  committed
  4aed3da

  Copy full SHA for 4aed3da

• Add TODO

  JCZuurmond

  committed
  183eba5

  Copy full SHA for 183eba5

• Add docstrings to methods

  JCZuurmond

  committed
  ace7ef2

  Copy full SHA for ace7ef2

• Add docstrings on MigrationStep attributes

  JCZuurmond

  committed
  ee42ec3

  Copy full SHA for ee42ec3

• Make types more consistent

  JCZuurmond

  committed
  3497b1c

  Copy full SHA for 3497b1c

• Make migration node frozen

  JCZuurmond

  committed
  f2324e8

  Copy full SHA for f2324e8

• Add generic to priorityqueue

  JCZuurmond

  committed
  7fcaaee

  Copy full SHA for 7fcaaee

• Fix PriorityQueue type hints

  JCZuurmond

  committed
  ad34395

  Copy full SHA for ad34395

• Update docs

  JCZuurmond

  committed
  c048880

  Copy full SHA for c048880

• Resolve TODO

  JCZuurmond

  committed
  4ba7408

  Copy full SHA for 4ba7408

• Add seen migration node set

  JCZuurmond

  committed
  9410c84

  Copy full SHA for 9410c84

• Use ordered steps to compute step number

  JCZuurmond

  committed
  08dfe32

  Copy full SHA for 08dfe32

• Fix create node queue

  JCZuurmond

  committed
  4b84652

  Copy full SHA for 4b84652

• Remove update from queue

  JCZuurmond

  committed
  c227165

  Copy full SHA for c227165

• Move mock admin locator to fixture

  JCZuurmond

  committed
  0138906

  Copy full SHA for 0138906

• Use ResourceDoesNotExists

  JCZuurmond

  committed
  74d115d

  Copy full SHA for 74d115d

• Introduce MaybeMigrationNode

  JCZuurmond

  committed
  12c6241

  Copy full SHA for 12c6241

• Test register job cluster

  JCZuurmond

  committed
  8b02db1

  Copy full SHA for 8b02db1

• Add happy path test for cluster

  JCZuurmond

  committed
  f12c25b

  Copy full SHA for f12c25b

• Rewrite order of workflow task to job

  JCZuurmond

  committed
  346f253

  Copy full SHA for 346f253

• Make register workflow task hidden

  JCZuurmond

  committed
  f301839

  Copy full SHA for f301839

• Use itertools.counter

  JCZuurmond

  committed
  f1225be

  Copy full SHA for f1225be

• Move register workflow job up

  JCZuurmond

  committed
  a4ac8fa

  Copy full SHA for a4ac8fa

• Rename to register job

  JCZuurmond

  committed
  0e07552

  Copy full SHA for 0e07552

• Make methods hidden

  JCZuurmond

  committed
  9b3b2e6

  Copy full SHA for 9b3b2e6

• Rename protected access methods

  JCZuurmond

  committed
  23d158b

  Copy full SHA for 23d158b

• Test job referencing unknown job cluster

  JCZuurmond

  committed
  844cc7f

  Copy full SHA for 844cc7f

• Test job with existing cluster

  JCZuurmond

  committed
  8261f0f

  Copy full SHA for 8261f0f

• Use MaybeMigrationNode when registering job

  JCZuurmond

  committed
  04d7fe5

  Copy full SHA for 04d7fe5

• Test register a job with non existing cluster

  JCZuurmond

  committed
  136ee2e

  Copy full SHA for 136ee2e

• Propagate cluster problems

  JCZuurmond

  committed
  8fc4437

  Copy full SHA for 8fc4437

• Update condition

  JCZuurmond

  committed
  ab40144

  Copy full SHA for ab40144

• Fix passing job node instead of job

  JCZuurmond

  committed
  5d93a77

  Copy full SHA for 5d93a77

• Remove redundant import

  JCZuurmond

  committed
  a6cbaad

  Copy full SHA for a6cbaad

• fix testing dependency problem

  JCZuurmond

  committed
  e8c64d3

  Copy full SHA for e8c64d3

• Add docstring to registering job

  JCZuurmond

  committed
  6a62ee0

  Copy full SHA for 6a62ee0

• Register job cluster

  JCZuurmond

  committed
  a29d130

  Copy full SHA for a29d130

• Test register job with non-existing job cluster key

  JCZuurmond

  committed
  6444363

  Copy full SHA for 6444363

• Test register existing job cluster

  JCZuurmond

  committed
  e93db45

  Copy full SHA for e93db45

• Fix missing parameter

  JCZuurmond

  committed
  a0b3a67

  Copy full SHA for a0b3a67

• Handle non-existing job cluster

  JCZuurmond

  committed
  4869679

  Copy full SHA for 4869679

• Test for new cluster

  JCZuurmond

  committed
  bc6f7f2

  Copy full SHA for bc6f7f2

• Fix sequencing

  JCZuurmond

  committed
  bea1eb1

  Copy full SHA for bea1eb1

• Fix parent object id

  JCZuurmond

  committed
  26b48c0

  Copy full SHA for 26b48c0

• Update sequencing

  JCZuurmond

  committed
  04e2943

  Copy full SHA for 04e2943

• Add type hints for problems

  JCZuurmond

  committed
  7e577e1

  Copy full SHA for 7e577e1

• Explain why not handling new cluster

  JCZuurmond

  committed
  d431702

  Copy full SHA for d431702

• Format

  JCZuurmond

  committed
  5f2d07b

  Copy full SHA for 5f2d07b

• Sequence with new cluster

  JCZuurmond

  committed
  8f85d69

  Copy full SHA for 8f85d69

• Add job cluster to nodes

  JCZuurmond

  committed
  d25664c

  Copy full SHA for d25664c

• Make job cluster id unique

  JCZuurmond

  committed
  698e338

  Copy full SHA for 698e338

• Make register signature similar

  JCZuurmond

  committed
  5551c26

  Copy full SHA for 5551c26

• Explain outgoing references

  JCZuurmond

  committed
  cb22042

  Copy full SHA for cb22042

• Remove TODO

  JCZuurmond

  committed
  cfc23c9

  Copy full SHA for cfc23c9

• Update docs and naming

  JCZuurmond

  committed
  6feebfa

  Copy full SHA for 6feebfa

• Update tests docs

  JCZuurmond

  committed
  ae059a0

  Copy full SHA for ae059a0

• Test sequence job with task referencing job cluster

  JCZuurmond

  committed
  2b90951

  Copy full SHA for 2b90951

• Fix test sequence in docstring

  JCZuurmond

  committed
  d453f53

  Copy full SHA for d453f53

• Add test when referencing non-existing cluster

  JCZuurmond

  committed
  5524191

  Copy full SHA for 5524191

• Remove unused fixtures

  JCZuurmond

  committed
  8d40f52

  Copy full SHA for 8d40f52

• Remove redundant comment and fix docstring

  JCZuurmond

  committed
  6d757bb

  Copy full SHA for 6d757bb

• Test task dependency

  JCZuurmond

  committed
  a4ace82

  Copy full SHA for a4ace82

• Sequence tasks with dependencies

  JCZuurmond

  committed
  742694f

  Copy full SHA for 742694f

• Add TODOs for register workflow task

  JCZuurmond

  committed
  f14a0f5

  Copy full SHA for f14a0f5

• Add todo for register cluster

  JCZuurmond

  committed
  d093a47

  Copy full SHA for d093a47

• Add migration sequencer to RuntimeContext

  JCZuurmond

  committed
  26f060b

  Copy full SHA for 26f060b

• Add integration test for simple job

  JCZuurmond

  committed
  418718f

  Copy full SHA for 418718f

• Test sequencing job with task referencing existing cluster

  JCZuurmond

  committed
  c55c821

  Copy full SHA for c55c821

• Test sequencing job with task referencing non existing cluster

  JCZuurmond

  committed
  23bec09

  Copy full SHA for 23bec09

• Format

  JCZuurmond

  committed
  b2c1671

  Copy full SHA for b2c1671

• Test sequence job with task dependency

  JCZuurmond

  committed
  0392cae

  Copy full SHA for 0392cae

• Test non-existing task dependency

  JCZuurmond

  committed
  0137e85

  Copy full SHA for 0137e85

• Return early when no job.settings

  JCZuurmond

  committed
  5278398

  Copy full SHA for 5278398

• Fix typo's

  JCZuurmond

  committed
  df45c53

  Copy full SHA for df45c53

• Rename administrator locator

  JCZuurmond

  committed
  1ea02f7

  Copy full SHA for 1ea02f7

• Do not expect cluster in sequence steps when not found

  JCZuurmond

  committed
  d26916b

  Copy full SHA for d26916b

• Refactor register_job to register_jobs

  JCZuurmond

  committed
  ba8b4c6

  Copy full SHA for ba8b4c6

• Remove unused import

  JCZuurmond

  committed
  9842f64

  Copy full SHA for 9842f64