ioztat is a storage load analysis tool for OpenZFS

Related tags

Miscellaneousioztat
Overview

ioztat

ioztat is a storage load analysis tool for OpenZFS. It provides iostat-like statistics at an individual dataset/zvol level.

The statistics offered are read and write operations per second, read and write throughput per second, and the average size (in kilobytes) of read and write operations issued in the current reporting interval. Viewing these statistics at the individual dataset level allows system administrators to identify storage "hot spots" in larger multi-tenant systems--particularly those with many VMs or containers operating essentially independent workloads.

This sample output shows activity which has taken place in the most recent second, on a the ssd zpool of a ZFS virtualization host:

[email protected]:~# ioztat -y -c1 ssd
dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
ssd                                            0.00       0.00       0.00       0.00       0.00       0.00
   images                                      0.00       0.00       0.00       0.00       0.00       0.00
      DC1                                     17.96       0.10       3.99       0.05       5.66      12.29
      DC2                                     21.95       0.14       0.00       0.00       6.59       0.00
      QB                                       2.00       0.01       0.00       0.00       7.17       0.00
      SAP-TC                                   2.99       0.02       0.00       0.00       6.83       0.00
      SAP4-WIN2019                             2.99       0.05       0.00       0.00      17.07       0.00
      nagios                                   0.00       0.00       0.00       0.00       0.00       0.00
      qemu                                     0.00       0.00       0.00       0.00       0.00       0.00
         autostart                             0.00       0.00       0.00       0.00       0.00       0.00
   iso                                         0.00       0.00       0.00       0.00       0.00       0.00
   unsnapped                                   0.00       0.00       0.00       0.00       0.00       0.00
      rp9                                      0.00       0.00       0.00       0.00       0.00       0.00

For the most part, ioztat behaves the same way that the system standard iostat tool does, with similar arguments.

usage: ioztat [-h] [-s SORT] [-i INTERVAL] [-c COUNT] [-y] [-b]
              pool [pool ...]

iostat for ZFS datasets

positional arguments:
  pool         ZFS pool

optional arguments:
  -h, --help   show this help message and exit
  -s SORT      Sort by: name / wps / wMBps / rps / rMBps
  -i INTERVAL  Time between each report
  -c COUNT     Number of reports generated
  -y           Skip the initial "summary" report
  -b           Use binary (power-of-two) prefixes

The only required argument is the name of at least one zpool to monitor. Without any other arguments, ioztat first prints a summary record showing activity per dataset since the most recent system boot, then prints a new record showing the most recent activity once per second. The -i argument can be used to change the report interval, and the -c argument can be used to limit ioztat to a certain number of intervals before exiting.

For those who wish a continually-updated, easy to read summary of pool activity, watch -n1 ioztat poolname -c1 -y will suit nicely.

Comments
  • Rework formatting

    Rework formatting

    This makes ioztat output more closely resemble zpool iostat:

                                  operations      bandwidth         opsize
    dataset                       read   write    read   write    read   write
    --------------------------  ------  ------  ------  ------  ------  ------
    rpool
       ROOT
          default                   23       0   62.6K       0    2.7K     759
       cvs                           0       0    1.6K     892   24.2K  108.4K
       home                          0       0       0       0       0       0
          freaky                     3       2   29.7K    8.3K   10.2K    3.9K
    

    This also adds -e for exact values (-p in zpool iostat, which we're currently using elsewhere) and -H for scripted mode that drops the header and separates fields by tabs.

    The intent is for this to eventually fix #23, probably by adjusting the space for dataset.

    opened by Freaky 12
  • Painting the bikeshed

    Painting the bikeshed

    Should bandwidth be changed to throughput? This is more correct, but deviates from zpool-iostat(8).

    Should we default to binary mode, since that's all zpool-iostat(8) supports?

    Should we default to a single iteration? This is what iostat(1) and zpool-iostat(8) both do. They also accept interval and count as positional arguments.

    Is + as an ellipsis for truncated dataset names reasonable, or should it be ... despite the additional space it uses?

    opened by Freaky 8
  • Overwrite old reports in terminal by default

    Overwrite old reports in terminal by default

    I thought that this script put a little too much into the terminal. This PR makes it so that only the most recent report is visible, which can be disabled with the new -n argument.

    opened by Aluminite 4
  • No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    ioztat -Hey as suggested by the man page doesn't return anything.

    ioztat -SIn rpool, also suggested by the man page, gives "dataset not found: 'rpool'" (rpool of course exists).

    Is there a way to further debug this? Am I missing a dependency?

    opened by steffen-AEI 3
  • Make numeric precision variable

    Make numeric precision variable

    Matching ZFS number formatting behaviour, use between 0 and 2 decimal places according to available space, and further reduce the column size to again match zpool-iostat and make more room for dataset names.

    This includes the careful rounding behaviour of the previous implementation, which I believe ZFS lacks.

    opened by Freaky 3
  • Fix -y when not combined with -c

    Fix -y when not combined with -c

    Fix an error when args.count is None. While I'm here also fix a count of zero that was ignored due to zero being falsey.

    Simplify the logic slightly by changing skipsummary from a bool to a generic skip count.

    opened by Freaky 3
  • Output is insensitive to terminal width

    Output is insensitive to terminal width

    The current output width is 106 columns, which by happy coincidence is precisely the width of my usual terminals.

    This coincidence seems unlikely to be universal, and some sensitivity to this would seem to be in order.

    opened by Freaky 3
  • Fix and simplify overwrite mode

    Fix and simplify overwrite mode

    This fixes overwrite mode by replacing the line counting and iterated cursor repositioning with a clear screen and placing the cursor at 1,1 - exactly what GNU watch does.

    I've also done some tidying, hoisting some iteration and filtering logic into dedicated generator functions which helps to further simplify the main loop.

    opened by Freaky 3
  • python3 shebang--does this work in FreeBSD?

    python3 shebang--does this work in FreeBSD?

    @Freaky, I discovered after merging your patches today that ioztat does not work on the Python2.7 that my Ubuntu workstation at the home office provides with "env python". I had to update the shebang to "env python3" to get it to work again.

    This should be fine for Ubuntu, but does it work in FreeBSD-land as well? I don't know if there's a python3 executable in your environment, or whether there is only python and you get what you get.

    opened by jimsalterjrs 3
  • Format name with all intermediates for disambiguation

    Format name with all intermediates for disambiguation

    Fixes #5

    This replaces the "just-the-rightmost-dataset-name-component" logic by "components-of-the-name-unique-after-the-previous-one". Output with default options looks like

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
    zroot                                          0.00       0.00       0.00       0.00       0.00       0.00
       ROOT/default                                0.00       0.00       3.88       0.00       0.00       0.16
       tmp                                         0.00       0.00       0.00       0.00       0.00       0.00
    

    (e.g. here instead of default being indented two, and ROOT being invisible, we get the dataset-name-components that distinguish it from zroot, the previously-printed name)

    opened by adriaandegroot 3
  • Unmounted datasets lead to confusing gaps in output

    Unmounted datasets lead to confusing gaps in output

    Stats for unmounted datasets aren't exported, so you can end up with incomplete hierarchies in ioztat output. For example here rpool and rpool/ROOT aren't mounted, leaving rpool/ROOT/default without the path leading up to it:

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
          default                                  0.00       0.00       0.97       0.00       0.00       0.05
    

    Similarly, rpool/usr/local/poudriere/ports/default misses the mountpoint-less poudriere/ports segments:

       usr                                         0.00       0.00       0.97       0.00       0.00       0.01
          local                                    0.00       0.00       0.00       0.00       0.00       0.00
                   default                         0.00       0.00       0.00       0.00       0.00       0.00
    
    opened by Freaky 3
  • using dataclasses

    using dataclasses

    Hi,

    I started of my own version of this, but found yours basically the last moment … My name of choice was objset, but the poll is finished I guess.

    I used dataclasses - ported here. And updated the fields to match current OpenZFS git.

    opened by commonism 7
  • no activity reported for zfs replication

    no activity reported for zfs replication

    is it expected that ioztat does not seem to report activity for zfs replication ?

    if this is not a bug but by design, maybe it's worth mentioning in the

    opened by devZer0 0
  • Packages

    Packages

    Produce .deb/.rpm/etc packages. sanoid is already doing this. I'm currently using sanoid package in latest ubuntu release.

    Getting these included will take some time but is worthwhile I think.

    enhancement help wanted 
    opened by morganchristiansson 2
Releases(v2.0.1)
  • v2.0.1(Mar 4, 2022)

  • v2.0.0(Mar 3, 2022)

    Added

    • An iostat(8) man page
    • CHANGELOG.md, this Changelog
    • -e and -H flags for use in scripting [#26]
    • -I flag for totals since last update instead of per-second
    • -o flag to overwrite prior reports [#9] [#20]
    • -P and -p flags to override dataset name display [#19]
    • -S flag for including child dataset statistics in parents [#32]
    • -T d|u flag for adding a timestamp to each report [#25]
    • -V, --version flags [#13] [#14]
    • -x flag for extended statistics, including unlink counts under -xx [#33]
    • interval and count positional arguments [#31]

    Changed

    • Shebang line is now /usr/bin/env python3 [#18]
    • Formatting has changed from resembling GNU iostat(8) to ZFS zpool-iostat(8) [#26]
    • Header is now printed periodically on a tty unless -N is specified [#26]
    • Display is clamped to the terminal width by truncating dataset names if necessary [#26]
    • Sort field names have been changed, with fallbacks for compatibility [#26]
    • Sort options are now case-insensitive [#26]
    • count now defaults to 1 unless an interval is specified [#31]
    • dataset is now an optional argument [#20]
    • Binary (1024-based) formatting is now default, with new -D flag for decimal
    • Average I/O sizes are now hidden beyind -x flag by default to reduce clutter [#33]
    • Exit with an error if a requested dataset does not exist or is not mounted

    Removed

    • -b flag. Binary mode is now the default to match other iostat tools.

    Fixed

    • count and interval must now be positive [#31]
    • WIFSIGNALED() status is now propagated properly to caller
    • Sleep interval now adjusts to compensate for runtime
    • Unhandled BrokenPipeError exception on SIGPIPE
    • Unhandled FileNotFoundError exception on Linux if a dataset is destroyed while enumerating files in /proc
    • Unhandled CalledProcessError exception on FreeBSD if a requested pool does not exist when using sysctl(8) fallback
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Jan 20, 2022)

    Thanks to the efforts of @Freaky and @adriaandegroot, ioztat has new features, improved output readability, and now works on FreeBSD as well as Linux!

    New features include initial selection at the dataset level rather than pool level, optional suppression of zero-activity datasets with -z, and optional suppression of recursion into child datasets with -n.

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 12, 2022)

    The first release of ioztat builds on efforts from the Reddit r/zfs community, including:

    u/55rzs (initial creation) u/d1722825 (substantial refactoring and cleanup) u/mercenary_sysadmin (addition of the -y flag to allow for easy use with the GNU watch command)

    Enjoy!

    Source code(tar.gz)
    Source code(zip)
Owner
Jim Salter
Jim Salter
OnTime is a small python that you set a time and on that time, app will send you notification and also play an alarm.

OnTime Always be OnTime! What is OnTime? OnTime is a small python that you set a time and on that time, app will send you notification and also play a

AmirHossein Mohammadi 11 Jan 16, 2022
Python bindings for Basler's VisualApplets TCL script generation

About visualapplets.py The Basler AG company provides a TCL scripting engine to automatize the creation of VisualApplets designs (a former Silicon Sof

Jürgen Hock 2 Dec 07, 2022
This script can be used to get unlimited Gb for WARP.

Warp-Unlimited-GB This script can be used to get unlimited Gb for WARP. How to use Change the value of the 'referrer' to warp id of yours You can down

Anix Sam Saji 1 Feb 14, 2022
Hacking and Learning consistently for 100 days straight af.

#100DaysOfHacking Hacking and Learning consistently for 100 days straight af. [yes, no breaks except mental-break ones, Obviously.] This Repo is one s

FENIL SHAH 17 Sep 09, 2022
Get you an ultimate lexer generator using Fable; port OCaml sedlex to FSharp, Python and more!

NOTE: currently we support interpreted mode and Python source code generation. It's EASY to compile compiled_unit into source code for C#, F# and othe

Taine Zhao 15 Aug 06, 2022
0CD - BinaryNinja plugin to introduce some quality of life utilities for obsessive compulsive CTF enthusiasts

0CD Author: b0bb Quality of life utilities for obsessive compulsive CTF enthusia

12 Sep 14, 2022
Mdisk - 🚧 On Construction 🚧

Mdisk Install For Package pip install mdisk pip install git+https://github.com/HeimanPictures/Mdisk.git Usage You can use this as python module or via

AkKiL 6 Aug 08, 2022
Cloud Native sample microservices showcasing Full Stack Observability using AppDynamics and ThousandEyes

Cloud Native Sample Bookinfo App Observability Bookinfo is a sample application composed of four Microservices written in different languages.

Cisco DevNet 13 Jul 21, 2022
A python script developed to process Windows memory images based on triage type.

Overview A python script developed to process Windows memory images based on triage type. Requirements Python3 Bulk Extractor Volatility2 with Communi

CrowdStrike 245 Nov 24, 2022
Margin Calculator - Personally tailored investment tool

Margin Calculator - Personally tailored investment tool

1 Jul 19, 2022
Get a link to the web version of a git-tracked file or directory

githyperlink Get a link to the web version of a git-tracked file or directory. Applies to GitHub and GitLab remotes (and maybe others but those are no

Tomas Fiers 2 Nov 08, 2022
A data driven app for bicycle hiring in London(UK)

bicycle_hiring_app_deployed A data driven app for bicycle hiring in London(UK). It predicts expected number of bicycle hire in London. It asks users t

Rajarshi Roy Raju 1 Dec 10, 2021
A (hopefully) considerably copious collection of classical cipher crackers

ClassicalCipherCracker A (hopefully) considerably copious collection of classical cipher crackers Written in Python3 (and run with PyPy) TODOs Write a

Stanley Zhong 2 Feb 22, 2022
Python package that mirrors the original Nodejs ReplAPI-It.

Python-ReplAPI-It Python package that mirrors the original Nodejs ReplAPI-It. Contributing First fork the repo: $ git clone https://github.com/ReplAPI

The ReplAPI.it Project 10 Jun 05, 2022
Python package for reference counting native pointers

refcount master: testing: This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boi

CSIRO Hydroinformatics 2 Nov 03, 2022
MDAnalysis tool to calculate membrane curvature.

The MDAkit for membrane curvature analysis is part of the Google Summer of Code program and it is linked to a Code of Conduct.

MDAnalysis 19 Oct 20, 2022
Extract continuous and discrete relaxation spectra from G(t)

pyReSpect-time Extract continuous and discrete relaxation spectra from stress relaxation modulus G(t). The papers which describe the method and test c

3 Nov 03, 2022
A Python Web Application for Checking vaccine slots by pincodes and auto slot booking.

The Dashboard is developed using Bokeh and python 3.5+. This dashboard is useful for you if you are looking for something which will help you to book the vaccine slot once slots become available. Oth

Suraj Deshmukh 10 Jan 23, 2022
A Linux program to create a Windows USB stick installer from a real Windows DVD or image.

WoeUSB-ng A Linux program to create a Windows USB stick installer from a real Windows DVD or image. This package contains two programs: woeusb: A comm

Longinus 1 Nov 19, 2021
PyDateWaiter helps waiting special day & calculating remain days till that day with Python code.

PyDateWaiter (v.Beta) PyDateWaiter helps waiting special day(aniversary) & calculating remain days till that day with Python code. Made by wallga gith

wallga 1 Jan 14, 2022