• Remove dependance on user home directory for temporary storage. User directories ("~/") are not always created for Unix service accounts.
• Create the temporary directory using tempfile.mkdtemp()
• Store the directory path in a singleton-like structure accessed via a function call

This fixes issue https://github.com/shaypal5/skift/issues/6 by creating the tempdir in an OS/environment agnostic way, and does not rely on the users' home directory being writeable.

Tried running this from the README:

pip install skift[fasttext] --process-dependency-links

Got this error:

Collecting fasttext==0.1.0+git.3b5fd29; extra == "fasttext" (from skift[fasttext])
  Could not find a version that satisfies the requirement fasttext==0.1.0+git.3b5fd29; extra == "fasttext" (from skift[fasttext]) (from versions: 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.5.1, 0.5.12, 0.5.13, 0.5.14, 0.5.15, 0.5.16, 0.5.17, 0.5.18, 0.5.19, 0.6.0, 0.6.1, 0.6.2, 0.6.4, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.8.0, 0.8.1, 0.8.2, 0.8.3)
 No matching distribution found for fasttext==0.1.0+git.3b5fd29; extra == "fasttext" (from skift[fasttext])

Tried with Python 3.6.4 in and out of a virtualenv. Seems skift expects to find a version of fasttext that's not available in pypi?

Hello!

I am trying to train a supervised text classification model on some text that contains also non-alphanumeric characters

from skift import FirstColFtClassifier
sk_clf = FirstColFtClassifier(lr=0.25, dim=100, epoch=100, minCount=5, 
                              minn=3, maxn=6, wordNgrams=3, loss='softmax')
sk_clf.fit(X_train, y_train)

As soon as the first non alphanumeric character occurs during training I get the following error

UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-8-05c208efc7be> in <module>()
      4                               minn=3, maxn=6, wordNgrams=3, loss='softmax')
      5 # Train fastText classifier
----> 6 sk_clf.fit(X_train, y_train)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\skift\core.py in fit(self, X, y)
    117         temp_trainset_fpath = temp_dataset_fpath()
    118         input_col = self._input_col(X)
--> 119         dump_xy_to_fasttext_format(input_col, y, temp_trainset_fpath)
    120         # train
    121         self.model = train_supervised(

~\AppData\Local\Continuum\anaconda3\lib\site-packages\skift\util.py in dump_xy_to_fasttext_format(X, y, filepath)
     68     with open(filepath, 'w+') as wfile:
     69         for text, label in zip(X, y):
---> 70             wfile.write('__label__{} {}\n'.format(label, text))
     71 
     72 

~\AppData\Local\Continuum\anaconda3\lib\encodings\cp1252.py in encode(self, input, final)
     17 class IncrementalEncoder(codecs.IncrementalEncoder):
     18     def encode(self, input, final=False):
---> 19         return codecs.charmap_encode(input,self.errors,encoding_table)[0]
     20 
     21 class IncrementalDecoder(codecs.IncrementalDecoder):

UnicodeEncodeError: 'charmap' codec can't encode character '\u010d' in position 493: character maps to <undefined>

As the error clearly shows, this is due to the fact that cp1252.py is the default encoder used by skift. Even though I am on a Windows OS, I am using Python 3.7 installed with Anaconda 5.3.0, and the standard encoding as far as I know should be UTF-8. (I have already verified that, by simply renaming the utf_8.py encoder as cp1252.py, the model training completes without any error. This is a dirty hack I would like to avoid though, because I plan to operationalize the model in production on Azure ML Studio).

Is there a way to enforce skift to use as default the utf_8.py encoder?

Any help appreciated!

Kind regards

The cli interface to fasttext to do parameter tuning and model quantization:

fasttext supervised -input model_train.train -output model_tune -autotune-validation model_train.valid -autotune-modelsize 100M -autotune-duration 1200 -loss one-vs-all

Do you plan to implement it in your package at some point ? If I can make a pr with a piece of code that does the job

Hi, added support for auto-tuning. Please LMK if you support this direction, and I'll add documentation and more tests to make it a mergeable PR.

Signed-off-by: Dimid Duchovny [email protected]

The functions predict, predict_proba return lists instead of numpy arrays which makes them unusable with classifiers like sklearn.multiclass.OneVsRestClassifier. GridSearch and other similar functionality also don't work.

This is a quick fix.

skift seems to expect integer labels and will fail when using string labels.

For instance, when running

from skift import FirstColFtClassifier
import pandas as pd
df = pd.DataFrame(
    data=[
        ['woof', 'a'],
        ['meow', 'b'],
        ['squick', 'c'],
    ],
    columns=['txt', 'lbl'],
)
sk_clf = FirstColFtClassifier(lr=0.3, epoch=10)
sk_clf.fit(df[['txt']], df['lbl'])
sk_clf.predict([['squick']])

I get

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-32-52a73258e761> in <module>
----> 1 sk_clf.predict([['squick']])

/usr/local/Caskroom/miniconda/base/envs/base/lib/python3.7/site-packages/skift/core.py in predict(self, X)
    165         return np.array([
    166             self._clean_label(res[0][0])
--> 167             for res in self._predict(X)
    168         ], dtype=np.float_)
    169 

/usr/local/Caskroom/miniconda/base/envs/base/lib/python3.7/site-packages/skift/core.py in <listcomp>(.0)
    165         return np.array([
    166             self._clean_label(res[0][0])
--> 167             for res in self._predict(X)
    168         ], dtype=np.float_)
    169 

/usr/local/Caskroom/miniconda/base/envs/base/lib/python3.7/site-packages/skift/core.py in _clean_label(ft_label)
    135     @staticmethod
    136     def _clean_label(ft_label):
--> 137         return int(ft_label[9:])
    138 
    139     def _predict_on_str_arr(self, str_arr, k=1):

ValueError: invalid literal for int() with base 10: 'c'

This is a bit unexpected since neither sklearn nor fasttext require integer labels.

I guess skift could handle that either by:

• passing the string labels directly to fasttext (caveat: might require some cleaning)
• automatically calling LabelEncoder (e.g. as in sklearn's code for LR)

fastText assumes UTF-8 encoded text (see fastText Python README).

Without the encoding flag, the xy input file is written using the system's locale, which is problematic, especially on Windows. Attempting to train a model with text which uses utf-8 symbols results in an exception.

Passing the flag to open when writing the input file solves this issue.

Hi,

I'm very sorry for asking such a basic question but can't work this one out! Usually, I see other text classifiers taking one of three forms;

1. (1D) List of strings, if it performs tokenisation and vectorisation itself
2. (2D) List of tokens if it performs vectorisation itself
3. (2D) List of vectors if it is just a classifier

I'm a little confused as the readme does not have a case where multiple tokens are inputted into the model. However, in the tests it appears is that it is trained on a pd.DataFrame for X and a pd.Series for y. I believe fasttext does the tokenisation and vectorisation itself, so why do we need a two dimensional input instead of a 1D list of strings? Is there benefit to doing it that way over something like this;

FtClassifier().fit(
    ['Input 1', 'Input 2'],
    [1, 0]
)

or the equivalent but with 1D numpy arrays?

Many thanks! Dom

Running skift in a docker container results in permission errors when trying to load previously generated models.

File "/usr/local/lib/python3.5/dist-packages/skift/util.py", line 10, in PermissionError: [Errno 13] Permission denied: '/root/.temp'

The problem is the docker container is running as user 'root', but the /root/ folder is not writable.

I have a fix, and will open a pull request shortly

how can we tune parameters? in https://fasttext.cc/docs/en/autotune.html uses autotuneValidationFile to feed validation see to model. how can we set this parameter?

Add support to providing multi-label labels in a scikit-learn-compliant format, utilizing (under the hood) fasttext's support for multi-label scenarios.