Precidition for all models produced during traing

[kozaka93]

mljar-supervised gives predictions only for the best model. Is there an option to return a prediction for all models produced during training?

[pplonski]

Hi @kozaka93,

Right now there is not such option. It would be nice to have such option. There is a similar issue for computing predictions for selected model #423

[giovannimicaroni]

Hi @pplonski, is this issue still open?

[pplonski]

Yes, it is still open.

[giovannimicaroni]

Great, me and @HenriquePManoel would like to work on it, if possible!

[pplonski]

Sure! Do you have plan how this will look like? Have you see #423?

[giovannimicaroni]

Not yet! We will structure a plan and look at the similar issue. Thank you

[giovannimicaroni]

Hi @pplonski, hope you are doing well. We structured an initial plan to solve this issue: Our idea is for the AutoML object constructor to have an attribute “models_mode”, for example, which is set to “best” by default, and would run the current MLJar implemented pipeline. (There could also be an option to choose the top x best models for more customizability, and set it to 1 as default. This could be done with another optional parameter in the constructor or in the fit step, for example "top_k_models")
However, we want to provide the user the option to set this attribute to “all”, to get predictions for all available models, and the option to provide a custom list with the exact models it wants to use for the predictions. We would need to change the fit and predict functions for this, correct? Do you think it is a viable solution? Thanks

[pplonski]

Hi @giovannimicaroni,

Thank you for proposition. By changing the constructor you force user to know how the predictions will be computed. I think this can be decided when computing predictions. What do you think?

[giovannimicaroni]

@pplonski That's true. However, i think that we would need to save all the models during the fit function if we were to let the user decide during prediction, right? I think it's doable, but another option could be for the user to decide during the fit function, so that we could save only the desired models and avoid using too much memory. What do you think?

[pplonski]

Right now we save all models, always. I wouldn't change this because it is safer to store all models on hard drive.

[giovannimicaroni]

Oh, I guess I missed that part of the code. Great, so we would only need to load the desired models during the predict functions and compute their predictions then? @pplonski

[pplonski]

Yes, we need to check is required models are not loaded, and load them if necessary.