Multi-output

[briandesilva]

Hi, very interesting package! I might be doing something wrong, but I also may have found a bug. The pyuoi linear models are subclasses of sklearn.base.MultiOutputMixin (i.e. isinstance(model, MultiOutputMixin) evaluates to True), but they don't appear to support multiple targets.

Minimal working example:

import numpy as np
from pyuoi.linear_model import UoI_ElasticNet

x = np.ones((5, 2))
model = UoI_ElasticNet()
model.fit(x, x)

Error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-41f22fb29892> in <module>
      3 x = np.ones((5, 2))
      4 model = UoI_ElasticNet()
----> 5 model.fit(x, x)

~/venv/lib/python3.6/site-packages/pyuoi/linear_model/base.py in fit(self, X, y, stratify, verbose)
    199             self._logger.setLevel(logging.WARNING)
    200 
--> 201         X, y = self._pre_fit(X, y)
    202 
    203         X, y = check_X_y(X, y, accept_sparse=['csr', 'csc', 'coo'],

~/venv/lib/python3.6/site-packages/pyuoi/linear_model/base.py in _pre_fit(self, X, y)
    538             if y.shape[1] > 1:
    539                 raise ValueError('y should either have shape ' +
--> 540                                  '(n_samples, ) or (n_samples, 1).')
    541         else:
    542             raise ValueError('y should either have shape ' +

ValueError: y should either have shape (n_samples, ) or (n_samples, 1).

[JesseLivezey]

@briandesilva, good catch. We're being a little sloppy with inheritance since we inherit from Elasticnet/Lasso but don't implement the multi-target versions.

There's potentially ~3 ways to implement the selection part of the multi-target UoI versions of these models

1. Enet/Lasso penalty with final selection profile shared across targets (through intersection)
2. Enet/Lasso penalty with final selection profile independent across targets
3. Group-Lasso penalty with final selection profile shared across targets

For your use, does one of these make the most sense?

We have 1 and 2 implemented for multiclass LogisticRegression. For Linear/Poisson models, I think 2 is almost equivalent to fitting the targets independently (up to exact Lasso path selection). Doing 2 jointly would require substantial work.

1 and 3 would require some re-working of the code, but should rely on existing sklearn models and so wouldn't be as difficult.

[briandesilva]

For my particular use-case option 2 makes the most sense because I expect that the targets will typically have different supports (I was playing around with using your linear models to carry out sparse regression in this package). The targets could be fit independently.

There's no rush on getting a fix out—I've implemented a simplified version of your union-of-intersections algorithm (literally just a pair of for loops) that should get the job done for now. However, if/when you put out a fix I'll add an example (and reference) showing how to use PyUoI regressors in combination with PySINDy.