Latent Semantic Analysis
Pipeline for training LSA models using Scikit-Learn.
Usage
Instead of writing custom code for latent semantic analysis, you just need:
- install pipeline:
pip install latent-semantic-analysis
- run pipeline:
- either in terminal:
lsa-train --path_to_config config.yaml
- or in python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
NOTE: more about config file here.
No data preparation is needed, only a csv file with raw text column (with arbitrary name).
Config
The user interface consists of only one files:
- config.yaml - general configuration with sklearn TF-IDF and SVD parameters
Change config.yaml to create the desired configuration and train LSA model with the following command:
- terminal:
lsa-train --path_to_config config.yaml
- python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
Default config.yaml:
seed: 42
path_to_save_folder: models
# data
data:
data_path: data/data.csv
sep: ','
text_column: text
# tf-idf
tf-idf:
lowercase: true
ngram_range: (1, 1)
max_df: 1.0
min_df: 1
# svd
svd:
n_components: 10
algorithm: arpack
NOTE: tf-idf and svd are sklearn TfidfVectorizer and TruncatedSVD parameters correspondingly, so you can parameterize instances of these classes however you want.
Output
After training the model, the pipeline will return the following files:
model.joblib- sklearn pipeline with LSA (TF-IDF and SVD steps)config.yaml- config that was used to train the modellogging.txt- logging filedoc2topic.json- document embeddingsterm2topic.json- term embeddings
Requirements
Python >= 3.6
Citation
If you use latent-semantic-analysis in a scientific publication, we would appreciate references to the following BibTex entry:
@misc{dayyass2021lsa,
author = {El-Ayyass, Dani},
title = {Pipeline for training LSA models},
howpublished = {\url{https://github.com/dayyass/latent-semantic-analysis}},
year = {2021}
}
9 Jul 17, 2021
42 Dec 20, 2022
190 Dec 21, 2022
7 Sep 20, 2022
57 Dec 7, 2022
74 Oct 7, 2022
6 Jun 27, 2022
654 Jan 2, 2023
14 Dec 9, 2022
2 Feb 17, 2022
161 Aug 07, 2022
11 Dec 09, 2022
0 Oct 07, 2021
99 Jan 06, 2023
118 Jan 06, 2023
8 Dec 25, 2022
71 Jan 04, 2023
3 Jan 14, 2022
66 Dec 26, 2022
43 Jan 05, 2023
16 Dec 11, 2022
72 Dec 19, 2022
7 Dec 25, 2022
3 Dec 14, 2022
98 Dec 09, 2022
16 Dec 07, 2022
261 Nov 12, 2022
387 Dec 30, 2022