Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed
• [X] Issue: closes #116

Description of the changes proposed in the pull request

• Add a new transformer feature_duplicator
• Add new meta decorator column_duplicatable to transformers and learners to duplicate columns

Background

Currently, the transformations (such as target_categorizer) replace the value in the column. A better approach would be to allow the user to preserve the original value, outputting the encoded feature to a new column. This would also enable users to apply more than one transformation to the same column (example: frequency and target encoding of the same feature) without needing to duplicate the column first.

Usage examples in comments.

Status

On Hold

Todo list

• [x] Issue: closes #109

Background context

There was a change on Shap's output (to a list of ndarray) on its newer versions, and the classification learners throw a ValueError when apply_shap=True for binary classification. Multiclass classification and regression are not affected.

Description of the changes proposed in the pull request

Uses shap_values[1] and explainer.expected_value[1] for binary classification.

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed

Background context

Add common causal evaluation techniques, such as cumulative effect curve and cumulative gain curve

Description of the changes proposed in the pull request

First, this PR adds a list of possible metrics that one can use as effects. Then, it uses those effects for causal model validation curves. Finally, it adds functions to compute the area under those causal inference curves

Where should the reviewer start?

1. Effects
2. Curves
3. AUC

Remaining problems or questions

Need to integrate the causal metrics with standard fklearn validators. This should be easy to do, as the causal AUC function all return a float.

Status

IN DEVELOPMENT

Todo list

• [ ] Documentation

Background context

As we add new models, this package gets bigger, what can lead to a bad user experience. So we want to find a way to enable user install only the dependencies he/she need

Description of the changes proposed in the pull request

We introduce different requirements file based on scope, so if you want to use only xgboost, you can install only it using pip install fklearn[xgboost], same for other models

Remaining problems or questions

• Are we ok with this change, or should we keep all_deps(except for test) as default?

Describe the feature and the current state.

Today we choose to install all packages required by fklearn, for development purposes this sounds good, but in production could be better to install only what you need. My idea is to use extra requirements, this way we can have something like:

pip install fklearn[xgboost]

And we only install the common packages + xgboost, instead of install everything. We should also include a fklearn[all] to install everything

Will this change a current behavior? How?

Yes, standalone install will not work with most models.

Additional Information

This needs to be well documented in order to avoid a bad user experience

Status

READY

Background context

On fklearn open source announcement meetup we did a live presentation about how to use fklearn

Description of the changes proposed in the pull request

Adding two notebooks:

• Notebook used to generate the dataset that was used on the presentation
• Notebook of the presentation with all the analyses/models done on top of the dataset

Where should the reviewer start?

Review both notebooks

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed
• [ ] Bump version
• [ ] Edit the changelog: ?

Background context

First attempt to implement an artificial data corruption feature. Sort of a Chaos Monkey for datasets.

Description of the changes proposed in the pull request

Implements a new perturbators.py file to store data perturbation functions and also a general function similar to validator in validator.py.

Related PRs

None.

Where should the reviewer start?

Macacaos.ipynb: simple demo of what was implemented.

perturbator.py: I tried to follow the same structure in validator.py: a general broad function perturbator() that receives specific column-wise perturbation functions like nullify() or shift_mu().

validator.py: added two methods chaos_validator and chaos_validator_iteration, analogous to their non chaos counterparts but with data corruption at either train-time or test-time.

Remaining problems or questions

Is creating the chaos_validator function the best approach? Maybe just an optional argument inside the original validator?

No logging in the new functions. How to do it?

Maybe more corruption functions.

Improve the demo. Current plot does not look that convincing on how the dataset is being degraded by nulls.

Tests.

Status

READY

• [ ] Issue: closes #112

Background context

quantile_biner would result in n+1 quantiles when the min or max values were equal to either the first or last quantile.

Description of the changes proposed in the pull request

Change values categorized as the '0' quantile to 1 and values categorizes as 'n+1' (n being the wanted number of quantiles) to n.

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed

Background context

Currently, in this project exists an only rank evaluator, the Spearman correlation that examines the rank order between two variables. However, if we must measure the quality of the two ranks given real and predicted scores, we have to consider using the Normalized Discount Cumulative Gain (NDCG). Here, there are two main motivations [1]

1. Highly relevant items are more useful when appearing earlier in a search – have higher ranks;
2. Highly relevant items are more useful than marginally relevant items, which are in turn more useful than non-relevant items.

This metric is widely used by recommender systems and information retrieval [2].

[1] https://en.wikipedia.org/wiki/Discounted_cumulative_gain.

[2] BOBADILLA, Jesús et al. Recommender systems survey. Knowledge-based systems, v. 46, p. 109-132, 2013.

Status

READY

Todo list

• [x] Tests added and passed
• [x] Issue: closes #109

Background context

Update shap to be compatible with latest shap versions

Description of the changes proposed in the pull request

Updated lgbm shap, and changed the way multiclass shap is calculated for catboost. UPD: added description for magic numbers

Where should the reviewer start?

src/fklearn/training/classification.py

Status

READY

Todo list

• [ ] Documentation
• [ ] Tests added and passed

Background context

Even though we have ROC AUC evaluator implemented in fklearn, we are still missing PR AUC evaluator.

Description of the changes proposed in the pull request

This PR:

1. Adds a new evaluator pr_auc_evaluator
2. Changes the generic auc evaluator name from auc_evaluator to roc_auc_evaluator

Instructions

• Follow the instructions in README.md
• Delete everything between parenthesis (...)
• Remove the sections that are not relevant

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed
• [x] Issue: closes #204

Background context

In the causal validation module and the curves file, it would be useful to add an ascending parameter for the cumulative effect and cumulative gain curves.

The current state is to order predictions descending:

ordered_df = df.sort_values(prediction, ascending=False).reset_index(drop=True) If we add an ascending: bool = False argument to the cumulative_effect_curve, cumulative_gain_curve, relative_cumulative_gain_curve, and effect_curves, a user could modify how these effects are computed, whether to do them ascending or descending by the prediction column.

Description of the changes proposed in the pull request

A model could output a prediction that is not necessarily positively related to the effect to be computed, so adding an option to order this relationship differently allows for effects and gains with negatively related predictions and outcomes to be computed adequately.

The changes are applied to curves.py and also on auc.py on the causal-effect module.

Where should the reviewer start?

Reviewing causal-effect/curves as there are the definition of the functions from which all ordering behavior is propagated.

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed

Background context

We are extending the fklearn capabilities with unsupervised algorithm support. First, we proposed to use k-means clustering implementation from scikit-learn since it is a simple and relevant unsupervised algorithm.

Description of the changes proposed in the pull request

• Add kmeans learner using scikit-learn K-means implementation

• Add Silhouette Coefficient to evaluate unsupervised results

• Add Davies-Bouldin score to evaluate unsupervised results

• Add generic_unsupervised_sklearn_evaluator method in order to provide support to scikit-learn unsupervised metrics

Where should the reviewer start?

kmeans_learner method at src/fklearn/training/unsupervised.py. Then, the unsupervised evaluation metrics at src/fklearn/validation/evaluators.py.

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed

(Nothing had to change in here)

Background context

Requiring a version of scikit-learn strictly lower tan 0.24 prevents the installation of fklearn in python 3.9. More context in this Slack thread.

Description of the changes proposed in the pull request

Versions of scikit-learn that can be installed in python 3.9 (and don't include any major change) are supported. The only breaking change is in IsolationForest, but a version check is added for backwards compatibility.

Where should the reviewer start?

Check the bump in the range of the scikit-learn supported versions and, if interested, check about the behaviour param in the IsolationForest's doc.

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed

Background context

To calculate causal effects by segments a quantile-based approach is used to create the segments. However, we've seen that this is usually not the ideal way to create them as there are methods that create segments that are more distinguishable between one another.

One such example is the Fisher-Jenks algorithm. A user could create its own partitioner with this algorithm like this:

import pandas as pd

from jenkspy import jenks_breaks
from toolz import curry
from typing import List


@curry
def fisher_jenks_partitioner(series: pd.Series, segments: int) -> List:
    
    bins = jenks_breaks(series, n_classes=segments)
    bins[0] = -float("inf")
    bins[-1] = float("inf")
    
    return bins

And use it in effect_by_segment:

from fklearn.causal.effects import linear_effect
from fklearn.causal.validation.curves import effect_by_segment

df = pd.DataFrame(dict(
    t=[1, 1, 1, 2, 2, 2, 3, 3, 3],
    x=[1, 2, 3, 1, 2, 3, 1, 2, 3],
    y=[1, 1, 1, 2, 3, 4, 3, 5, 7],
))

result = effect_by_segment(
    df,
    prediction="x",
    outcome="y",
    treatment="t",
    segments=3,
    effect_fn=linear_effect,
    partition_fn=fisher_jenks_partitioner)

Or use another custom partitioner such as:

@curry
def bin_partitioner(series: pd.Series, segments: int = 1) -> List:
    return [1, 4, 5]

Description of the changes proposed in the pull request

We're adding:

• an argument to the effect_by_segment function so a user can define the way the segments are created.
• the quantile_partitioner so the default behavior of effect_by_segment is maintained.
• a new PartitionFnType type.
• tests for quantile_partitioner
• documentation for the new fklearn.causal.partitioners module

Related PRs

NA

Where should the reviewer start?

At the modifications we did in effect_by_segment and then to the quantile_partitioner definition.

Remaining problems or questions

We are not creating additional partitioners to the ones used by default because this would require more complex definitions or imports on new libraries (such as the Fisher-Jenks algorithm).

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed
• [x] ~Issue: closes NA~

Background context

We're adding a MinMax scaler to fklearn.training module, often used in neural networks.

Description of the changes proposed in the pull request

We've made two changes:

1. Adding the MinMax scaler to fklearn.training module.
2. Adding tests of the MinMax scaler.

Related PRs

NA

Where should the reviewer start?

With the definition of the MinMax scaler itself.

Remaining problems or questions

NA

Status

READY

Todo list

• [x] Documentation
• [x] Tests added and passed

Background context

This is a correlation-based feature selection method. But unlike the already existing correlation_feature_selection which does not have a criteria to selected among correlated features, feature_clustering_selection first employs a feature clustering, using absolute correlation as distance metric, following by the selection of the feature with lower 1-R2 metric from each cluster. 1-R2 metric allows to find the feature that most preserve the information (own cluster R2) from the other features from the same clusters, penalizing by the information (nearest cluster R2) present in the nearest cluster.

Description of the changes proposed in the pull request

This commit will add the feature selection method feature_clustering_selection in fklearn/tuning/model_agnostic_fc.py

Where should the reviewer start?

The reviewer should start by method feature_clustering_selection at src/fklearn/tuning/model_agnostic_fc.py The method test_feature_clustering_selection at fklearn/tests/tuning/test_model_agnostic_fc.py illustrates how is the method usage.