Simpler UI

[stufraser1]

We need to make the UI as simple as possible for the user.
Minimise number of times user has to specify class, header, quoting, runner. Assume these are common to all files in one conversion. Default as CSV / pandas.

May need to still have 6 path fields (extractor and loader for each of loc, ACC, ri) but can we minimise to point to one folder which should contain all files?

Transformation of acc, loc, RI files can run separately in back.

Validation should be reported for each filetype separately - each on own tab?

[OmegaDroid]

My thought here is that we could update the config so that we have a something like:

template:
   <the existing config structure>
transformations:
   <label>:
        <overrides>
...

The overrides would be a partial set of configuration fields that change the template. We can then add a set of templates to the tool which can be selected from the UI. In the case where we have ACC, LOC and RI CSV files this may look like:

template:
  mapping:
    options:
      input_format: A
      output_format: B
  extractor:
    options:
      quoting: nonnumeric
      write_header: True
  loader:
    options:
      quoting: nonnumeric
      write_header: True
transformations:
   ACC:
    extractor:
      options:
        path: ACC_in.csv
    loader:
      options:
        path: ACC_out.csv
   LOC:
    extractor:
      options:
        path: LOC_in.csv
    loader:
      options:
        path: LOC_out.csv
   RI:
    extractor:
      options:
        path: RI_in.csv
    loader:
      options:
        path: RI_out.csv

Which would mean we could then hide the template behind an advanced settings box or something as well as any fields already set in the template.

We still have an issue that we would still need to specify the mapping for each transformation as a different format would need to be selected for each transformation. We could solve this by reworking the mappings slightly so that rather than specifying a string as the input and output formats we would instead specify an object that includes class and subclass so rather than having AIR_ACC we would have:

input_format:
  class: AIR
  subclass: ACC

We would then use the labels from the transformations config to select the correct subclass (unless the config explicitly overrides it).

We would also want to think about whether we want to support the existing formats or move everything to the new (NOTE: the actual transformations aren't going to change, just the format metadata and any existing configs). We could support both but if the current usage is all development it may be easier to drop the old formats now rather than later.

There is also a smaller piece of work here that would be worth doing before the rest which is running multiple transformations from one config which would involve moving the config to:

transformations:
   <label>:
        <overrides>

And running them in the same command

[OmegaDroid]

@stufraser1 Do you think there will be any need for mixing formats in a single config? I can see people may want to do multiple transformations in one run but it seems fairly edge case and may be something we don't want to allow out of convention.

If we don't allow it (ie all transforms will be from CEDE -> OED but all file types are handled in a single file) the configuration will become simpler.

[stufraser1]

I think its reasonable to limit it to a single format in any single run