Asynchronous, fast, pythonic DynamoDB Client

Overview

AsyncIO DynamoDB

CircleCI Code style: black Documentation Status

Asynchronous pythonic DynamoDB client; 2x faster than aiobotocore/boto3/botocore.

Quick start

With httpx

Install this library

pip install "aiodynamo[httpx]" or, for poetry users poetry add aiodynamo -E httpx

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.httpx import HTTPX
from httpx import AsyncClient

    async with AsyncClient() as h:
        client = Client(HTTPX(h), Credentials.auto(), "us-east-1")

With aiohttp

Install this library

pip install "aiodynamo[aiohttp]" or, for poetry users poetry add aiodynamo -E aiohttp

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession

    async with ClientSession() as session:
        client = Client(AIOHTTP(session), Credentials.auto(), "us-east-1")

API use

        table = client.table("my-table")

        # Create table if it doesn't exist
        if not await table.exists():
            await table.create(
                Throughput(read=10, write=10),
                KeySchema(hash_key=KeySpec("key", KeyType.string)),
            )

        # Create or override an item
        await table.put_item({"key": "my-item", "value": 1})
        # Get an item
        item = await table.get_item({"key": "my-item"})
        print(item)
        # Update an item, if it exists.
        await table.update_item(
            {"key": "my-item"}, F("value").add(1), condition=F("key").exists()
        )

Why aiodynamo

  • boto3 and botocore are synchronous. aiodynamo is built for asynchronous apps.
  • aiodynamo is fast. Two times faster than aiobotocore, botocore or boto3 for operations such as query or scan.
  • aiobotocore is very low level. aiodynamo provides a pythonic API, using modern Python features. For example, paginated APIs are automatically depaginated using asynchronous iterators.
  • Legible source code. botocore and derived libraries generate their interface at runtime, so it cannot be inspected and isn't typed. aiodynamo is hand written code you can read, inspect and understand.
  • Pluggable HTTP client. If you're already using an asynchronous HTTP client in your project, you can use it with aiodynamo and don't need to add extra dependencies or run into dependency resolution issues.

Complete documentation is here

Comments
  • question: Any chance for transaction support?

    question: Any chance for transaction support?

    I'm wondering if there are any plans to support TransactWriteItems or just Transactions in general.

    I can probably work on this if/when I have more time 🤔

    opened by stupoid 16
  • add AWS_PROFILE env var to credentials chain

    add AWS_PROFILE env var to credentials chain

    Adds support for using AWS_PROFILE env var in the credentials chain.
    This behavior is more in line with the behavior expected by users coming from boto3 or aiobotocore. Added a unit test to cover this feature.

    Relates to: Issue #139

    opened by BTripp1986 8
  • Http wrapping issues

    Http wrapping issues

    When I was adding credentials handler to assume role with web identity which we need for our EKS setup I came across few issues why I am for now implementing it as custom code in our own project instead of providing it for the library.

    Http wrappers assume response is json, https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py but as endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs 💩 ). This is blocker and thought that perhaps there needs to be some refactoring around http handling which takes too long for us.

    Some additional things I noticed

    1. in MetadataCredentials there is fetch_with_retry which couldn't use because it assumes GET but endpoint for assume role is POST. Was thinking should/could this retry option be in base http classes?

    2. Missing timeout in POST, would prefer to have timeout also for this credentials call. As mentioned in https://github.com/HENNGE/aiodynamo/issues/45

    3. http.post(..) requires body as bytes even when it can in aiohttp and httpx also be None. https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py#L28-L30

    opened by jarikujansuu 6
  • Add support for pay_per_request billing mode during table creation

    Add support for pay_per_request billing mode during table creation

    Currently aiodynamo creates tables using the provisioned billing mode. DynamoDB also supports pay_per_request where no throughput config is provided. This PR is backwards compatible, but also adds the support to provide a billing_mode parameter to create_table. The throughput parameter has been changed to optional as the pay_per_request billing mode doesn't use it.

    PS. I'd be very grateful if you could also add the hacktoberfest-accepted label to this PR once it is accepted so that it can be counted as a hacktoberfest PR. Thanks :)

    hacktoberfest-accepted 
    opened by TheEdgeOfRage 6
  • Handle ClientConnectorError in connector

    Handle ClientConnectorError in connector

    Exception text: ClientConnectorError: Cannot connect to host dynamodb.<snip>.amazonaws.com:443 ssl:default [None]

    Exception type: aiohttp.client_exceptions.ClientConnectorError

    Stack:

    aiohttp/connector.py in _wrap_create_connection at line 943
    aiohttp/connector.py in _create_direct_connection at line 980
    aiohttp/connector.py in _create_direct_connection at line 1004
    aiohttp/connector.py in _create_connection at line 858
    aiohttp/connector.py in connect at line 523
    aiohttp/client.py in _request at line 480
    aiohttp/client.py in __aenter__ at line 1012
    aiodynamo/http/aiohttp.py in post at line 31
    aiodynamo/client.py in send_request at line 651
    aiodynamo/client.py in get_item at line 454
    aiodynamo/client.py in get_item at line 124
    
    opened by dimaqq 6
  • Raise asyncio.TimeoutError on httpx.TimeoutException

    Raise asyncio.TimeoutError on httpx.TimeoutException

    Now asyncio.TimeoutError is handled in the client logic. But httpx wrapper raises httpx.TimeoutException. This pull request convert the exception and make the client be able to handle timeouts in httpx.

    ref.) httpx timeout exceptions https://github.com/encode/httpx/blob/master/httpx/_exceptions.py#L7-L11

    ref.) Custom HTTP Client Adaptor document https://github.com/HENNGE/aiodynamo/blob/2e6c4716a3ac9fe5669bbdcaa73e6bbe0f73cfbb/docs/advanced.rst

    opened by gunyarakun 5
  • Count limits

    Count limits

    This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

    opened by Tinche 5
  • Use 3rd-party ddbcereal library for deserialize/serialize.

    Use 3rd-party ddbcereal library for deserialize/serialize.

    Figured I'd put this out here as an option. I've been working on a serializer/deserializer lib to supplement my use of aiobotocore, but it might make sense for this project too. It should be as fast or faster than this one.

    Overall, it reduces/simplifies the code, but the initialization of Client is a tad uglier due to ddbcereal needing a pre-constructed Deserializer and Client being frozen. It could be made pretty by unfreezing it or alternatively, making a seperate client-spawning function that passes the Deserializer to the Client constructor.

    opened by JustinTArthur 5
  • Don't send `StreamSpecification: null`

    Don't send `StreamSpecification: null`

    https://github.com/HENNGE/aiodynamo/blob/81b6f98111ce4bed41aeb8d07747695af836c76a/src/aiodynamo/client.py#L41-L49

    Sends StreamSpecification=None when none is supplied by the caller. Perhaps aiodynamo should not inject this key if the value is None.

    It seems that's OK for AWS SaaS dynamo, dynamodb-local and dynalite, but not ScyllaDB:

    https://github.com/scylladb/scylla/issues/5796

    opened by dimaqq 5
  • How to update map element using a pattern?

    How to update map element using a pattern?

    Described in https://dev.to/matthewvielkind/updating-values-in-dyanmodb-map-attributes-can

    UpdateExpression="SET players.#player_id.score = :score_val",  
    ExpressionAttributeNames={"#player_id": player_id},  
    

    I'm not sure what the API is in this case... Would I need to overload F?

    opened by dimaqq 4
  • Support consistent read

    Support consistent read

    Purpose

    Support consistent read in get_item, query, scan, count and batch get operations.

    Note: Tests do not actually check anything except that DynamoDb calls do not explode if using consistent_read option. Local DynamoDb is always strongly consistent. And might want to reduce copied code in them.

    opened by jarikujansuu 4
  • Empty-set safety

    Empty-set safety

    aiodynamo used to be empty-string-safe when DynamoDB didn't support empty string values. This mercifully got fixed in DynamoDB, but DynamoDB still doesn't accept empty sets (NS, SS, BS). There's an argument to be made that aiodynamo should handle this automatically, converting F("some_set").set(set()) to F("some_set").remove(), possibly with a warning.

    opened by ojii 3
  • Shall we try goodfirstissue.dev ?

    Shall we try goodfirstissue.dev ?

    links: https://goodfirstissue.dev/language/python https://github.com/deepsourcelabs/good-first-issue#adding-a-new-project

    I think this project may qualify, and would get some exposure and helping hands

    opened by dimaqq 0
  • Support AWS Web Identity Token File authentication

    Support AWS Web Identity Token File authentication

    aiodynamo ~~doesn't support AWS Web Identity Token File authentication~~ doesn't support AWS Web Identity Token File authentication out of the box.

    It looks like a typical case. For my team, we would like to use that authentication in our services in k8s by using IAM Roles for Service Accounts(IRSA)

    Related issue: https://github.com/HENNGE/aiodynamo/issues/128 Related PR: https://github.com/HENNGE/aiodynamo/pull/127

    There was a similar issue, but I decided to create another one because I think we can describe our goal and issue more directly.

    So, as discovered in a related PR, we should implement the AssumeRoleWithWebIdentity API for this authentication.

    opened by mrkovalchuk 2
  • Auth Error With No Default Credentials

    Auth Error With No Default Credentials

    I have access to several AWS accounts via SSO and no default profile for AWS. When I sign in to AWS via SSO the .aws/credentials file is updated with a fresh Access Key, Secret Key, and Security Token.

    I specify which credentials I will use with the AWS_PROFILE env var, which is checked as part of the credentials chain lookup. If using boto3 or the aws-cli this method works. With this method I can sign in via SSO to dev and prod accounts. Then if I AWS_PROFILE=dev aws s3 ls(or similar call in boto3) it will run the command against the dev account. Then if I AWS_PROFILE=prod aws s3 ls it will run against prod.

    This doesn't seem to work with aiodynamo. In order to get it to work I need to create a default profile and then copy/paste the credentials from the desired profile into the default profile. Aiodynamo should respect the AWS_PROFILE env var.

    opened by BTripp1986 1
  • Surprising behaviour in `Client.table_exists`

    Surprising behaviour in `Client.table_exists`

    We use Table.exists as part of a health check for the containers in our application, which in turn calls Client.table_exists. Today we had a short outage when all health checks began failing. It turned out the Dynamo table had entered UPDATING status.

    The implementation does the following:

    async def table_exists(self, name: TableName) -> bool:
            try:
                description = await self.describe_table(name)
            except TableNotFound:
                return False
    
            return description.status is TableStatus.active
    

    This feels like a bug to us: UPDATING is a valid state while a variety of operations are being applied to the table. So the code should do return description.status in [TableStatus.active, TableStatus.updating].

    Thoughts?

    opened by darylweir 1
Releases(21.9)
  • 21.9(Sep 1, 2021)

  • 21.8(Aug 24, 2021)

  • 21.7(Aug 24, 2021)

  • 21.6(Aug 24, 2021)

  • 21.5(Aug 24, 2021)

  • 20.4.2(Apr 21, 2020)

    Release Date: April 15th, 2020

    • Fix comparison conditions (equals, not_equals, gt, gte, lt, lte on aiodynamo.expressions.F and aiodynamo.expressions.Size via aiodynamo.expressions.F.size() to support referencing other fields (using aiodynamo.expressions.F)
    • Fix timeout handling in aiohttp based client.
    Source code(tar.gz)
    Source code(zip)
  • 20.4.1(Apr 21, 2020)

    Release Date: April 13th, 2020

    • Fixed put_item and delete_item with a condition which does not carry any values.
    • Wrap underlying HTTP client errors, such as connection issues, so networking issues during requests are retried.
    Source code(tar.gz)
    Source code(zip)
  • 20.4(Apr 21, 2020)

    Release Date: April 3rd, 2020

    • Fixed scan with a projection but no filter_expression.
    • Fixed logs leaking session tokens (request sending) and keys (metadata fetch).
    Source code(tar.gz)
    Source code(zip)
  • 20.3(Apr 21, 2020)

    Release Date: March 31st, 2020

    • Added TTL support
    • Added support for pluggable HTTP clients. Built in support for httpx and aiohttp.
    • Added custom client implementation.
    • Added custom credentials loaders, with support for custom credential loaders.
    • Fixed a typo in delete_item
    • Improved item deserialization performance
    • Improved overall client performance, especially for query, scan and count, which are now up to twice as fast.
    • Changed condition, key condition and filter expression APIs to not rely on boto3.
    • Moved aiodynamo.models.F to aiodynamo.expressions.F.
    • Removed boto3 dependency
    • Removed botocore dependency
    • Removed aiobotocore dependency
    Source code(tar.gz)
    Source code(zip)
  • 19.9(Apr 21, 2020)

  • 19.3(Mar 6, 2019)

  • 19.1(Jan 22, 2019)

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

AWS Data Wrangler Pandas on AWS Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretMana

Amazon Web Services - Labs 3.3k Dec 31, 2022
Use SQL query in a jupyter notebook!

SQL-query Use SQL query in a jupyter notebook! The table I used can be found on UN Data. Or you can just click the link and download the file undata_s

Chuqin 2 Oct 05, 2022
#crypto #cipher #encode #decode #hash

🌹 CYPHER TOOLS 🌹 Written by TMRSWRR Version 1.0.0 All in one tools for CRYPTOLOGY. Instagram: Capture the Root 🖼️ Screenshots 🖼️ 📹 How to use 📹

50 Dec 23, 2022
MinIO Client SDK for Python

MinIO Python SDK for Amazon S3 Compatible Cloud Storage MinIO Python SDK is Simple Storage Service (aka S3) client to perform bucket and object operat

High Performance, Kubernetes Native Object Storage 582 Dec 28, 2022
pandas-gbq is a package providing an interface to the Google BigQuery API from pandas

pandas-gbq pandas-gbq is a package providing an interface to the Google BigQuery API from pandas Installation Install latest release version via conda

Google APIs 348 Jan 03, 2023
sync/async MongoDB ODM, yes.

μMongo: sync/async ODM μMongo is a Python MongoDB ODM. It inception comes from two needs: the lack of async ODM and the difficulty to do document (un)

Scille 428 Dec 29, 2022
Find graph motifs using intuitive notation

d o t m o t i f Find graph motifs using intuitive notation DotMotif is a library that identifies subgraphs or motifs in a large graph. It looks like t

APL BRAIN 45 Jan 02, 2023
A Redis client library for Twisted Python

txRedis Asynchronous Redis client for Twisted Python. Install Install via pip. Usage examples can be found in the examples/ directory of this reposito

Dorian Raymer 127 Oct 23, 2022
SpyQL - SQL with Python in the middle

SpyQL SQL with Python in the middle Concept SpyQL is a query language that combines: the simplicity and structure of SQL with the power and readabilit

Daniel Moura 853 Dec 30, 2022
A Telegram Bot to manage Redis Database.

A Telegram Bot to manage Redis database. Direct deploy on heroku Manual Deployment python3, git is required Clone repo git clone https://github.com/bu

Amit Sharma 4 Oct 21, 2022
Simple Python demo app that connects to an Oracle DB.

Cloud Foundry Sample Python Application Connecting to Oracle Simple Python demo app that connects to an Oracle DB. The app is based on the example pro

Daniel Buchko 1 Jan 10, 2022
A Python wheel containing PostgreSQL

postgresql-wheel A Python wheel for Linux containing a complete, self-contained, locally installable PostgreSQL database server. All servers run as th

Michel Pelletier 71 Nov 09, 2022
CouchDB client built on top of aiohttp (asyncio)

aiocouchdb source: https://github.com/aio-libs/aiocouchdb documentation: http://aiocouchdb.readthedocs.org/en/latest/ license: BSD CouchDB client buil

aio-libs 53 Apr 05, 2022
A simple wrapper to make a flat file drop in raplacement for mongodb out of TinyDB

Purpose A simple wrapper to make a drop in replacement for mongodb out of tinydb. This module is an attempt to add an interface familiar to those curr

180 Jan 01, 2023
Asynchronous, fast, pythonic DynamoDB Client

AsyncIO DynamoDB Asynchronous pythonic DynamoDB client; 2x faster than aiobotocore/boto3/botocore. Quick start With httpx Install this library pip ins

HENNGE 48 Dec 18, 2022
Python client for Apache Kafka

Kafka Python client Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the offici

Dana Powers 5.1k Jan 08, 2023
Baserow is an open source no-code database tool and Airtable alternative

Baserow is an open source no-code database tool and Airtable alternative

1.3k Jan 01, 2023
Generate database table diagram from SQL data definition.

sql2diagram Generate database table diagram from SQL data definition. e.g. "CREATE TABLE ..." See Example below How does it works? Analyze the SQL to

django-cas-ng 1 Feb 08, 2022
A Pythonic, object-oriented interface for working with MongoDB.

PyMODM MongoDB has paused the development of PyMODM. If there are any users who want to take over and maintain this project, or if you just have quest

mongodb 345 Dec 25, 2022
Dlsite-doujin-renamer - Dlsite doujin renamer tool with python

dlsite-doujin-renamer Features 支持深度查找带有 RJ 号的文件夹 支持手动选择文件夹或拖拽文件夹到软件窗口 支持在 config

111 Jan 02, 2023