Fix dynamic nodes stuck in COMPLETING after their instance is terminated and there is a new instance reusing the same IP
Source: github.com/aws/aws-parallelcluster/issues/7671
Summary
A dynamic node that still has a job can get stuck as IDLE+CLOUD+COMPLETING+POWER_DOWN after clustermgtd terminates its EC2 instance. The job never finishes completing, the node is never powered down, and clustermgtd takes no further action.
Root cause

clustermgtd finds the node unhealthy, terminates its instance and runs scontrol update state=power_down_force, which also resets NodeAddr to the node name.
POWER_DOWN_FORCE kills the job but leaves the node’s base state alone. Slurm still treats the node as reachable and waits for EPILOG_COMPLETE. SuspendProgram doesn’t run while the node is COMPLETING.
Normally slurmd stops answering, and after SlurmdTimeout Slurm marks the node DOWN and recovers it. But if EC2 gives the old private IP to another compute node, that node’s slurmd answers in its place. So the node keeps looking alive, and the epilog never completes.
clustermgtd never notices. It decides whether a node should have an instance from NodeAddr, and NodeAddr has already been reset. Slurm still holds the old InstanceId, because it only clears InstanceId at POWERED_DOWN.

Change
Add SlurmNode.is_instance_expected() and use it in is_backing_instance_valid(). A node is now expected to have an instance if NodeAddr is set, or if InstanceId is set and the node isn’t powering down or powered down. Slurm keeps InstanceId until POWERED_DOWN, so a stuck node is now reported as missing its instance. clustermgtd then treats it as unhealthy and sends another power_down_force. For a job that is already completing, that clears the node’s epilog wait (kill_running_job_by_node_ptr), so the jobs are requeued and the node powers down.
Scope

Detection logic only; the scontrol commands clustermgtd sends are unchanged.
We considered setting nodes DOWN before power_down_force, which releases jobs immediately. It’s deferred because it changes behavior for every power-down caller.