This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
Extended functionality for Namebase past their web UI

Namebase Extended Extended functionality for Namebase past their web UI.

RunDavidMC 12 Sep 02, 2022
Macros in Python: quasiquotes, case classes, LINQ and more!

MacroPy3 1.1.0b2 MacroPy is an implementation of Syntactic Macros in the Python Programming Language. MacroPy provides a mechanism for user-defined fu

Li Haoyi 3.2k Jan 06, 2023
Let's pretend you want to create a AWS Lambda project called "sns-processor".

Usage Let's pretend you want to create a AWS Lambda project called "sns-processor". Rather than using lambda and then editing the results to include y

1 Dec 31, 2021
A collection of examples of using cocotb for functional verification of VHDL designs with GHDL.

At the moment, this repo is in an early state and serves as a learning tool for me. So it contains a a lot of quirks and code which can be done much better by cocotb-professionals.

T. Meissner 7 Mar 10, 2022
Simple web application, which has a single endpoint, dedicated to annotation parsing and convertion.

Simple web application, which has a single endpoint, dedicated to annotation parsing and conversion.

Pavel Paranin 1 Nov 01, 2021
Versión preliminar análisis general de Covid-19 en Colombia

Covid_Colombia_v09 Versión: Python 3.8.8 1/ La base de datos del Ministerio de Salud (Minsalud Colombia) está en https://www.datos.gov.co/Salud-y-Prot

Julián Gómez 1 Jan 30, 2022
Verification of Monty Hall problem by experimental simulation.

Verification of Monty Hall problem by experimental simulation. |中文|English| In the process of learning causal inference, I learned about the Monty Hal

云端听茗 1 Nov 22, 2022
A very small (15 lines of code) and beautiful fetch script (exclusively for Arch Linux).

minifetch A very small (15 lines of code) and beautiful fetch script (exclusively for Arch Linux). There are many fetch scripts out there but I wanted

16 Jul 11, 2022
The only purpose of a byte-sized application is to help you create .desktop entry files for downloaded applications.

Turtle 🐢 The only purpose of a byte-sized application is to help you create .desktop entry files for downloaded applications. As of usual with elemen

TenderOwl 14 Dec 29, 2022
The ldapconsole script allows you to perform custom LDAP requests to a Windows domain

ldapconsole The ldapconsole script allows you to perform custom LDAP requests to a Windows domain. Features Authenticate with password Authenticate wi

Podalirius 38 Dec 09, 2022
Automator anble you to create automations on your system

WELCOME TO AUTOMATOR BETA This programm is able to create automations on your system. This programm is only an experimantal release; infact it works v

Davide 1 Jan 12, 2022
A python script that will automate the boring task of login to the captive portal again and again

A python script that will automate the boring task of login to the captive portal again and again

Rakib Hasan 2 Feb 09, 2022
This script is written with Python for selling steam community items automatically.

SteamCommunityItemAutoSell Description This script is written with Python for selling steam community items automatically. Install git clone https://g

14 Oct 26, 2022
A Python3 script to decode an encoded VBScript file, often seen with a .vbe file extension

vbe-decoder.py Decode one or multiple encoded VBScript files, often seen with a .vbe file extension. Usage usage: vbe-decoder.py [-h] [-o output] file

John Hammond 147 Nov 15, 2022
ABT aka Animated Background Tool is a windows only python program that makes it that you can have animated background.

ABT ABT aka Animated Background Tool is a windows only python program that makes it that you can have animated background. 𝓡𝓔𝓐𝓓 𝓜𝓔, An Important

Yeeterboi4 2 Jul 16, 2022
You'll learn about Iterators, Generators, Closure, Decorators, Property, and RegEx in detail with examples.

07_Python_Advanced_Topics Introduction 👋 In this tutorial, you will learn about: Python Iterators: They are objects that can be iterated upon. In thi

Milaan Parmar / Милан пармар / _米兰 帕尔马 252 Dec 23, 2022
Chat meetup

FLiP-Meetup-Chat Chat meetup create function bin/pulsar-admin functions create --auto-ack true --jar pulsardjlexample-1.0.jar --classname "dev.pulsarf

Timothy Spann 1 Dec 09, 2021
A demo Piccolo app - a movie database!

PyMDb Welcome to the Python Movie Database! Built using Piccolo, Piccolo Admin, and FastAPI. Created for a presentation given at PyData Global 2021. R

11 Oct 16, 2022
BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset.

BloodCheck BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset. Installation

Mr B0b 16 Nov 05, 2021
Python most simple|stupid programming language (MSPL)

Most Simple|Stupid Programming language. (MSPL) Stack - Based programming language "written in Python" Features: Interpretate code (Run). Generate gra

Kirill Zhosul 14 Nov 03, 2022