Luminaire is a python package that provides ML driven solutions for monitoring time series data.

The current approach for trend detection in the Data exploration module (/luminaire/exploration/data_exploration.py) was enabled only for daily ('D') and hourly ('H) time series. I added a fix to trigger the computation of window sizes for weekly ('W') frequency, using a value of 4. Also, I added a fix to support all the other frequencies.

• This is related to issue Weekly data exploration failure #112

• This PR is to replace the PR https://github.com/zillow/luminaire/pull/113

We met the problem when we tried to call the score function in WindowDensity API (Luminaire Libary). The error message was "setting an array element with a sequence". We searched online and asked for other professionals' experience but still failed to solve it. Can anybody help us with it? Thanks in advance~~

Luminaire Reference: https://zillow.github.io/luminaire/_modules/luminaire/model/window_density.html#WindowDensityHyperParams

Hey all!

I'm trying to use the package but I'm getting that message.

import luminaire
import pandas as pd

from luminaire.exploration.data_exploration import DataExploration

past = pd.read_csv("dataset.csv").set_index("index")

de = DataExploration(freq='D')

past_prof, profile = de.profile(df=past)
#(None,
#{'success': False,
# 'ErrorMessage': "unsupported operand type(s) for -: 'int' and 'NoneType'"})

Is that anything data-related?

Here are my infos:

• Python 3.7.10
• requirements.txt: see below, result from pip install -U jupyterlab numpy pandas matplotlib luminaire pip setuptools pyarrow

Thanks!

anyio==3.6.1
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
attrs==22.1.0
Babel==2.10.3
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==5.0.1
boto3==1.24.76
botocore==1.27.76
certifi==2022.9.14
cffi==1.15.1
changepy==0.3.1
charset-normalizer==2.1.1
cloudpickle==2.2.0
cycler==0.11.0
debugpy==1.6.3
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
fastjsonschema==2.16.2
fonttools==4.37.2
future==0.18.2
hyperopt==0.2.7
idna==3.4
importlib-metadata==4.12.0
importlib-resources==5.9.0
ipykernel==6.15.3
ipython==7.34.0
ipython-genutils==0.2.0
jedi==0.18.1
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
json5==0.9.10
jsonschema==4.16.0
jupyter-core==4.11.1
jupyter-server==1.18.1
jupyter_client==7.3.5
jupyterlab==3.4.7
jupyterlab-pygments==0.2.2
jupyterlab_server==2.15.1
kiwisolver==1.4.4
luminaire==0.4.0
lxml==4.9.1
MarkupSafe==2.1.1
matplotlib==3.5.3
matplotlib-inline==0.1.6
mistune==2.0.4
nbclassic==0.4.3
nbclient==0.6.8
nbconvert==7.0.0
nbformat==5.5.0
nest-asyncio==1.5.5
networkx==2.6.3
notebook==6.4.12
notebook-shim==0.1.0
numpy==1.21.6
packaging==21.3
pandas==1.3.5
pandas-redshift==2.0.5
pandocfilters==1.5.0
parso==0.8.3
patsy==0.5.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.2.0
pkgutil_resolve_name==1.3.10
prometheus-client==0.14.1
prompt-toolkit==3.0.31
psutil==5.9.2
psycopg2-binary==2.9.3
ptyprocess==0.7.0
py4j==0.10.9.7
pyarrow==9.0.0
pycparser==2.21
Pygments==2.13.0
pykalman==0.9.5
pyparsing==3.0.9
pyrsistent==0.18.1
python-dateutil==2.8.2
pytz==2022.2.1
pyzmq==24.0.0
requests==2.28.1
s3transfer==0.6.0
scikit-learn==1.0.2
scipy==1.7.3
Send2Trash==1.8.0
six==1.16.0
sniffio==1.3.0
soupsieve==2.3.2.post1
statsmodels==0.13.2
terminado==0.15.0
threadpoolctl==3.1.0
tinycss2==1.1.1
tomli==2.0.1
tornado==6.2
tqdm==4.64.1
traitlets==5.4.0
typing_extensions==4.3.0
urllib3==1.26.12
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.4.1
zipp==3.8.1

The current approach for trend turning was enabled only for daily and hourly time series. Added a quick fix to trigger computation of window sizes for other frequency types.

Corrected issue where the diff order was hard coded as 2 in lad_filtering. Also added test_lad_filtering_scoring_diff_order to test_models which uses the last data points, takes the appropriate diff, and then compares to the adjusted actual to make sure the appropriate diff order is applied.

Related Issue: #120 @sayanchk for review

I guess python 3.7 and later considered not supported (see https://github.com/zillow/luminaire/runs/1946332964)

On python 3.6 pyramid-arima wheel build will fail (but it will not affect the installation of dependency - just produce log noise) without a numpy installed, but it looks like it's not required to actually have it as dependecy - see https://github.com/zillow/luminaire/pull/74

P.S. https://pip.pypa.io/en/latest/reference/pip_install/#controlling-setup-requires There is a warning about how dangerous to use this keyword, but i guess it's ok for such simple case It's also used in https://github.com/zillow/luminaire/pull/77/

No actual content change in the documentation.

• Replaced the incomplete sphinx theme with a more polished one, along with corresponding stylesheets
• Fixed some indentation issues in the docs
• Shuffled files around: removed dedicated TOC pages and added them all in the index instead

Screenshot of the home page:

Here's a second screenshot that shows syntax highlighting and footer (closes #40)

@sayanchk you might want to look into shortening the page titles for the API ref (or just name them after the modules)

I have tried to use simple data and its giving these issues

Here is the notebook https://colab.research.google.com/drive/19muQTHoWxdh5fC1DQE2FpYu763fn-0zC?usp=sharing

exit 1 will will called only if first flake8 will fail and return non zero code from script block immediately

Before last command of script block was evaluated and second flake8 invocation was always returning 0 because of flag passed

I think it will be worth to have local means of running tests/lint even if you have CI perfectly working (python setup.py test, python setup.py flake8) I used config from https://github.com/zillow/luminaire/blob/master/.github/workflows/python-app.yml#L42 for flake Flake gonna nuke the integration at some point, but i guess it's ok for now (we can use last version without this warning - it's not that old)

https://gitlab.com/pycqa/flake8/-/issues/544

Hi there,

I have a question rather than any specific issues. I wonder if this library can work with missing points/date during training stage? and what about anomaly detection at seconds level data? I will appreciate your response

When diff_order is applied in lad_filtering.py, the value passed to np.diff is fixed as 2. Is this intended or should diff_order be passed instead?

if diff_order:
  actual_previous_per_diff = [interpolated_actual_previous[-1]] \
      if diff_order == 1 else [interpolated_actual_previous[-1], np.diff(interpolated_actual_previous)[0]]
  seq_tail = interpolated_actual_previous + [interpolated_actual]
  interpolated_actual = np.diff(seq_tail, 2)[-1]

_detect_window_size is currently not optimized for weekly time series data in order to detect the most frequent periodic pattern. This issue need some investigation on that front. Reference: https://github.com/zillow/luminaire/pull/114

Note: This method is a dependency for Structural, Filtering and Window based models. Therefore, any change in this method requires testing on any existing supported (or rather optimized on) time series data types (daily, hourly and even higher frequencies). Please refer to the datasets for testing.

The repo has linter running but there are quite some warnings which are not breaking but needs to be resolved.

Example pipeline: https://github.com/zillow/luminaire/runs/5104474237?check_suite_focus=true

11    C901 'DataExploration._detrender' is too complex (13)
7     E122 continuation line missing indentation or outdented
12    E127 continuation line over-indented for visual indent
29    E128 continuation line under-indented for visual indent
2     E[203](https://github.com/zillow/luminaire/runs/5104474237?check_suite_focus=true#step:6:203) whitespace before ':'
2     E225 missing whitespace around operator
2     E231 missing whitespace after ','
3     E266 too many leading '#' for block comment
22    E302 expected 2 blank lines, found 1
10    E303 too many blank lines (2)
9     E501 line too long (134 > 127 characters)
1     E714 test for object identity should be 'is not'
2     E722 do not use bare 'except'
16    F401 'luminaire.optimization' imported but unused
5     F403 'from luminaire.exploration.data_exploration import *' used; unable to detect undefined names
34    F405 'DataExploration' may be undefined, or defined from star imports: luminaire.exploration.data_exploration
1     F841 local variable 'e' is assigned to but never used
1     W291 trailing whitespace
3     W292 no newline at end of file
4     W293 blank line contains whitespace
1     W391 blank line at end of file

Hello!

Is there any way to extract the dataframes containing the decomposition of the time series? That is, one column for the trend, another for the seasonality, etc.

Thanks

Hello I have the following data frame.

I am calling it using imputed_data, pre_prc = de_obj.profile(hourly, impute_only=True)

and getting the following error. {'success': False, 'ErrorMessage': "ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''"}

I have been trying to figure it out, but no avail. Any help would be much appreciated. Thanks!