Image build fails because of undefined cluster_compute_groups

[priteau]

Packer image build fails because it refers to variable cluster_compute_groups defined by OpenTofu.

    openstack.openhpc: TASK [osc.ood : Install OOD apps] **********************************************                                                                                                                                                                                                                                                                     
    openstack.openhpc: task path: /home/rocky/image-build/ansible/roles/osc.ood/tasks/install-apps.yml:1                                                                                                                                                                                                                                                                    
    openstack.openhpc: The conditional check 'ood_install_apps is defined' failed. The error was: error while evaluating conditional (ood_install_apps is defined): {{ ood_install_apps_defaults if openondemand_jupyter_partition | default(none) else {} }}: {{ openhpc_partitions[0]['name'] }}: {{ openhpc_user_partitions + ([openhpc_rebuild_partition] if groups['reb
uild'] | length > 0 else []) }}: {{ openhpc_nodegroups }}: {{ cluster_compute_groups | map('community.general.dict_kv', 'name') }}: 'cluster_compute_groups' is undefined                                                                                                                                                                                                   
    openstack.openhpc:                                                                                                                                                                
    openstack.openhpc: The error appears to be in '/home/rocky/image-build/ansible/roles/osc.ood/tasks/install-apps.yml': line 1, column 3, but may
    openstack.openhpc: be elsewhere in the file depending on the exact syntax problem.                                                                                                
    openstack.openhpc:                                                                                                                                                                                                                                                                                                                                                      
    openstack.openhpc: The offending line appears to be:                                                                                                                              
    openstack.openhpc:                                                                                                                                                                                                                                                                                                                                                      
    openstack.openhpc:                                                                                                                                                                
    openstack.openhpc: - name: Install OOD apps                                                                                                                                                                                                                                                                                                                             
    openstack.openhpc:   ^ here                                                       
    openstack.openhpc: fatal: [default]: FAILED! => {}                                                                                                                                
    openstack.openhpc:                                                                                                                                                                                                                                                                                                                                                      
    openstack.openhpc: MSG:                                                                                                                                                           
    openstack.openhpc:                                                                                                                                                                                                                                                                                                                                                      
    openstack.openhpc: {{ ood_install_apps_defaults if openondemand_jupyter_partition | default(none) else {} }}: {{ openhpc_partitions[0]['name'] }}: {{ openhpc_user_partitions + ([openhpc_rebuild_partition] if groups['rebuild'] | length > 0 else []) }}: {{ openhpc_nodegroups }}: {{ cluster_compute_groups | map('community.general.dict_kv', 'name') }}: 'cluster_
compute_groups' is undefined                                                                                                                                                                                                                                                                                                                                                
    openstack.openhpc:                                                                                                                                                                

[sjpb]

Oof. So the train is:

# environments/common/inventory/group_vars/all/openondemand.yml
ood_install_apps ->
ood_install_apps_defaults ->
openondemand_jupyter_partition ->
openondemand_jupyter_partition ->
# environments/common/inventory/group_vars/all/openhpc.yml
openhpc_partitions -> openhpc_user_partitions
openhpc_nodegroups -> cluster_compute_groups

which, as you say is defined in tf so is undefined normally. Need to work out why this works for CI builds.

[priteau]

I used this workaround:

diff --git a/environments/common/inventory/group_vars/all/openhpc.yml b/environments/common/inventory/group_vars/all/openhpc.yml
index 005bdf04..003d2b77 100644
--- a/environments/common/inventory/group_vars/all/openhpc.yml
+++ b/environments/common/inventory/group_vars/all/openhpc.yml
@@ -27,7 +27,7 @@ openhpc_rebuild_partition: # not a role var - could actually add more indirectio
     DisableRootJobs: NO
     PreemptMode: 'OFF'
     OverSubscribe: EXCLUSIVE
-openhpc_nodegroups: "{{ cluster_compute_groups | map('community.general.dict_kv', 'name') }}" # create nodegroup for each compute group
+openhpc_nodegroups: "{{ cluster_compute_groups | default([{'name': 'standard'}]) | map('community.general.dict_kv', 'name') }}" # create nodegroup for each compute group
 openhpc_user_partitions: "{{ openhpc_nodegroups }}" # create partition for each nodegroup (actually role default) - this is what we'd expect to be changed
 openhpc_partitions: "{{ openhpc_user_partitions + ([openhpc_rebuild_partition] if groups['rebuild'] | length > 0 else []) }}" # auto-create rebuild partition if reqd.
 openhpc_packages_default:

[sjpb]

Ah here's why it works for CI image builds:

#environments/.stackhpc/inventory/group_vars/all/openondemand.yml
openondemand_jupyter_partition: standard

[sjpb]

I don't think defaulting openhpc_nodegroups like that is correct, its messing at the wrong level and there is no default that you have a nodegroup named "standard".

The correct thing to do is to set openondemand_jupyter_partition to the partition you want to use for jupyter. I'm not sure how else we can get the desired behaviour of "jupyter installs by default" and "if you don't turn on jupyter you don't get it installed". But actually at the momemt making openondemand_jupyter_partition falsy is not obvious either.

[sjpb]

Thinking about it, the production docs should call out setting openondemand_jupyter_partition and openondemand_desktop_partition, the defaults are not expected to be correct for "production" clusters. I think any better fix than that needs the planned wholesale overhaul of how we produce new environments.

[sjpb]

I guess it does say "configure Open Ondemand" and points to the specific docs. But they say those *_partition vars have defaults, which might look fine. Maybe we should just make installing jupyter the default, and you empty ood_install_apps if you don't want to install things.