Supervised domain-agnostic prediction framework for probabilistic modelling

Re-opening the sub-issue opened in #3 and commented upon by @murphyk

Question: should skpro's predict methods return a vector of distribution objects? For example, using the distributions from scipy.stats which implement methods pdf, cdf, mean, var, etc.

Pro:

• this would be using an existing, consolidated, and well-supported interface
• it might be easier to use
• it might be easier to understand

Contra:

• mixture types are not supported
• l2 norm is not supported (as would be needed for squared/Gneiting loss)
• mixed distributions on the reals, especially empirical distributions (weighted sum of deltas) which are returned by Bayesian packages are not supported
• vectors of distributions are not supported, alternatively Cartesian products of distributions
• this is not the status quo

I'm following along with this intro example.. However this line fails

(numpy.mean(y_pred) * 2).shape

Error below (seems to be because Distribution objects don't support the mean() function but instead insist on obscurely calling it point!)

np.mean(y_pred)
Traceback (most recent call last):

  File "<ipython-input-38-19819be87ab5>", line 1, in <module>
    np.mean(y_pred)

  File "/home/kpmurphy/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2920, in mean
    out=out, **kwargs)

  File "/home/kpmurphy/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py", line 75, in _mean
    ret = umr_sum(arr, axis, dtype, out, keepdims)

TypeError: unsupported operand type(s) for +: 'Distribution' and 'Distribution'

The first example in your documentation (DensityBaseline) does not run right on my machine: it throws a 'module not found' exception at the call to 'utils'.

This might be a python version problem (I am using 3.6), so perhaps it's not an error in the normal sense - though I don't see any specification that the package required a particular python version. Apologies if I missed it: in any case, I fixed it by importing matplotlib instead: i.e.

import matplotlib.pyplot as plt plt.scatter(y_test, y_pred)

instead of:

import utils utils.plot_performance(y_test, y_pred)

It has been 2 days that I am trying to import skpro. But I can not I keep getting this error:

cannot import name 'six' from 'sklearn.externals' (C:\Users\My Book\anaconda3\lib\site-packages\sklearn\externals_init_.py)

A wishlist for probabilistic regression methods to implement or interface. This is partly copied from the R counterpart https://github.com/mlr-org/mlr3proba/issues/32 . Number of stars at the end is estimated difficulty or time investment.

GLM

• [ ] generalized linear model(s) with regression link, e.g., Gaussian *
• [ ] generalized linear model(s) with count link, e.g., Poisson *
• [ ] heteroscedastic linear regression ***
• [ ] Bayesian GLM where conjugate priors are available, e.g., GLM with Gaussian link ***

KRR aka Gaussian process regression

• [ ] vanilla kernel ridge regression with fixed kernel parameters and variance *
• [ ] kernel ridge regression with MLE for kernel parameters and regularization parameter **
• [ ] heteroscedastic KRR or Gaussian processes ***

CDE

• [ ] variants of conditional density estimation (Nadaraya-Watson type) **
• [ ] reduction to density estimation by binning of input variables, then apply unconditional density estimation **

Tree-based

• [ ] probabilistic regression trees **

Neural networks

• [ ] interface tensorflow probability - some hard-coded NN architectures **
• [ ] generic tensorflow probability interface - some hard-coded NN architectures ***

Bayesian toolboxes

• [ ] generic pymc3 interface ***
• [ ] generic pyro interface ****
• [ ] generic Stan interface ****
• [ ] generic JAGS interface ****
• [ ] generic BUGS interface ****
• [ ] generic Bayesian interface - prior-valued hyperparameters *****

Pipeline elements for target transformation

• [ ] distr fixed target transformation **
• [ ] distr predictive target calibration **

Composite techniques, reduction to deterministic regression

• [ ] stick mean, sd, from a deterministic regressor which already has these as return types into some location/scale distr family (Gaussian, Laplace) *
• [ ] use model 1 for the mean, model 2 fit to residuals (squared, absolute, or log), put this in some location/scale distr family (Gaussian, Laplace) **
• [ ] upper/lower thresholder for a regression prediction, to use as a pipeline element for a forced lower variance bound **
• [ ] generic parameter prediction by elicitation, output being plugged into parameters of a distr object not necessarily scale/location ****
• [ ] reduction via bootstrapped sampling of a determinstic regressor **

Ensembling type pipeline elements and compositors

• [ ] simple bagging, averaging of pdf/cdf **
• [ ] probabilistic boosting ***
• [ ] probabilistic stacking ***

baselines

• [ ] always predict a Gaussian with mean = training mean, var = training var *
• [ ] IMPORTANT as featureless baseline: reduction to distr/density estimation to produce an unconditional probabilistic regressor **
• [ ] IMPORTANT as deterministic style baseline: reduction to deterministic regression, mean = prediction by det.regressor, var = training sample var, distr type = Gaussian (or Laplace) **

Other reduction from/to probabilistic regression

• [ ] reducing deterministic regression to probabilistic regression - take mean, median or mode **
• [ ] reduction(s) to quantile regression, use predictive quantiles to make a distr ***
• [ ] reducing deterministic (quantile) regression to probabilistic regression - take quantile(s) **
• [ ] reducing interval regression to probabilistic regression - take mean/sd, or take quantile(s) **
• [ ] reduction to survival, as the sub-case of no censoring **
• [ ] reduction to classification, by binning ***

See below some comments/description of the coming refactoring contents :

• Distribution classes refactoring in a more OOD way (see. skpro->distribution)
• Losse functions (see. metrics->distribution)
• Estimators (see. metrics->distribution)

Some descriptive notebooks (in docs->notebooks) and a full set of unit test (in tests) are also available.