Determine how to automatically identify Partitions (e.g. gaia) max wall time dynamically

[lichtefeld]

If we can find the maxwall time information dynamically at runtime then we can simplify the users' declaration of a partition to only be a str of the partition name rather than needing to provide a job max wall time specifically. (e.g. we can default to the maximum amount of time with a warning)

@danielnapierski suggests investigating:

How could we do this better? By calling sacctmgr to get live values?

[joecummings]

I'm not sure I understand. If no job time is provided, we already do default to the maximum amount of time with a warning:
[2021-04-01 08:31:26] WARNING:root:Defaulting job with no job time specified to max walltime of selected partition 'ephemeral'

[lichtefeld]

Correct we already pull values from this dict. If you try to make a job with the aro-batts partition and don't specify a wall-time we raise an error as we don't know what the max-wall-time of aro-batts is.

The hope is rather than needing to maintain a dictionary that is always in parallel to the partitions on the cluster to be able to determine these values at run-time of the python script.

[joecummings]

aro-batts max walltime is the same as the DEFAULT_WALLTIME, which should be returned when it can't find the aro-batts name in that dict. Is this a bug?