Avoid PyCharm Overloading
=========================
This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.
On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.
Why?
----
When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).
Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.
How?
----
To avoid stalling these processes, we modify their sources as follows:
1. modify the conda cli
$ conda activate base
$ pip install runfast
$ nano `which conda`
Modify the main section:
if __name__ == '__main__':
from conda.cli import main
from runfast import cached
cached(main)
2. modify the conda_packaging_tool.py
if __name__ == '__main__':
from runfast import cached
cached(main)
How does this work?
-------------------
Two simple steps:
1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
given the same command line parameters
That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.
How to clear the cache or avoid caching at all?
$ export RUNFAST_NOCACHE=1
Testing
-------
a) conda packaging tool
# first time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m13.271s
user 0m9.631s
sys 0m1.936s
# second time
$ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
real 0m3.159s
user 0m0.139s
sys 0m1.298s
b) conda cli
# first time
$ time conda list -p /path/to/env
real 0m7.985s
user 0m7.883s
sys 0m0.087s
# second time
real 0m0.152s
user 0m0.116s
sys 0m0.032s
References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
This speeds up PyCharm's package index processes and avoids CPU & memory overloading
Overview
Extended functionality for Namebase past their web UI
Namebase Extended Extended functionality for Namebase past their web UI.
Macros in Python: quasiquotes, case classes, LINQ and more!
MacroPy3 1.1.0b2 MacroPy is an implementation of Syntactic Macros in the Python Programming Language. MacroPy provides a mechanism for user-defined fu
Let's pretend you want to create a AWS Lambda project called "sns-processor".
Usage Let's pretend you want to create a AWS Lambda project called "sns-processor". Rather than using lambda and then editing the results to include y
A collection of examples of using cocotb for functional verification of VHDL designs with GHDL.
At the moment, this repo is in an early state and serves as a learning tool for me. So it contains a a lot of quirks and code which can be done much better by cocotb-professionals.
Simple web application, which has a single endpoint, dedicated to annotation parsing and convertion.
Simple web application, which has a single endpoint, dedicated to annotation parsing and conversion.
Versión preliminar análisis general de Covid-19 en Colombia
Covid_Colombia_v09 Versión: Python 3.8.8 1/ La base de datos del Ministerio de Salud (Minsalud Colombia) está en https://www.datos.gov.co/Salud-y-Prot
Verification of Monty Hall problem by experimental simulation.
Verification of Monty Hall problem by experimental simulation. |中文|English| In the process of learning causal inference, I learned about the Monty Hal
A very small (15 lines of code) and beautiful fetch script (exclusively for Arch Linux).
minifetch A very small (15 lines of code) and beautiful fetch script (exclusively for Arch Linux). There are many fetch scripts out there but I wanted
The only purpose of a byte-sized application is to help you create .desktop entry files for downloaded applications.
Turtle 🐢 The only purpose of a byte-sized application is to help you create .desktop entry files for downloaded applications. As of usual with elemen
The ldapconsole script allows you to perform custom LDAP requests to a Windows domain
ldapconsole The ldapconsole script allows you to perform custom LDAP requests to a Windows domain. Features Authenticate with password Authenticate wi
Automator anble you to create automations on your system
WELCOME TO AUTOMATOR BETA This programm is able to create automations on your system. This programm is only an experimantal release; infact it works v
A python script that will automate the boring task of login to the captive portal again and again
A python script that will automate the boring task of login to the captive portal again and again
This script is written with Python for selling steam community items automatically.
SteamCommunityItemAutoSell Description This script is written with Python for selling steam community items automatically. Install git clone https://g
A Python3 script to decode an encoded VBScript file, often seen with a .vbe file extension
vbe-decoder.py Decode one or multiple encoded VBScript files, often seen with a .vbe file extension. Usage usage: vbe-decoder.py [-h] [-o output] file
ABT aka Animated Background Tool is a windows only python program that makes it that you can have animated background.
ABT ABT aka Animated Background Tool is a windows only python program that makes it that you can have animated background. 𝓡𝓔𝓐𝓓 𝓜𝓔, An Important
You'll learn about Iterators, Generators, Closure, Decorators, Property, and RegEx in detail with examples.
07_Python_Advanced_Topics Introduction 👋 In this tutorial, you will learn about: Python Iterators: They are objects that can be iterated upon. In thi
Chat meetup
FLiP-Meetup-Chat Chat meetup create function bin/pulsar-admin functions create --auto-ack true --jar pulsardjlexample-1.0.jar --classname "dev.pulsarf
A demo Piccolo app - a movie database!
PyMDb Welcome to the Python Movie Database! Built using Piccolo, Piccolo Admin, and FastAPI. Created for a presentation given at PyData Global 2021. R
BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset.
BloodCheck BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset. Installation
Python most simple|stupid programming language (MSPL)
Most Simple|Stupid Programming language. (MSPL) Stack - Based programming language "written in Python" Features: Interpretate code (Run). Generate gra