[python-package] fix misleading redundant parameter warnings in Booster.refit()

[arjunprakash027]

Fix Misleading Redundant Parameter Warnings in Booster.refit()

Fixes #6793

Problem

As discussed in #6793 , the Booster.refit() method raises misleading warnings about redundant parameters being passed to the internal Dataset constructor. The issue occurs when parameters like categorical_feature or label_column are stored in the Booster's internal params dictionary and then inadvertently passed both as keyword arguments and within the params dictionary to Dataset(), triggering false-positive warnings.

For additional context, see #6793 (comment)

Solution

This PR modifies Booster.refit() to implement parameter routing:

1. Parameter Inspection: Uses inspect.signature() to dynamically identify which parameters belong to the Dataset constructor by examining Dataset._lazy_init.

2. Parameter Routing: Checks if a Dataset-related parameter (such as categorical_feature or label_column) exists in the Booster's internal params but has not been explicitly overridden by the user in the refit() call. When a parameter qualifies for routing (i.e., it's a Dataset parameter with a default value in the refit() signature), it is moved from new_params to the local variable scope for the Dataset constructor. This ensures each parameter is passed exactly once to the internal Dataset object, eliminating the redundant warning.

I've tried to resolve the warning issue without changing the actual behavior of the refitting process.

[jameslamb]

Thanks for all the investigation you've done so far! Leaving a blocking review, as I'd like the opportunity to look into this and suggest a different fix.

This patch with a double-nested if block inside a for loop and uses of inspect and locals() looks quite complex, and I'm worried it'd be difficult to modify correctly in the future.

[arjunprakash027]

Thanks @jameslamb!
I agree, The complexity of code I wrote is overkill for this fix. I can try and write a simple fix, and also investigate if the problem is booster.refit() specific or if its an general problem.
Or, I can wait for your suggestion for a different fix.