Javascript image annotation tool based on image segmentation.

Overview

JS Segment Annotator

Javascript image annotation tool based on image segmentation.

  • Label image regions with mouse.
  • Written in vanilla Javascript, with require.js dependency (packaged).
  • Pure client-side implementation of image segmentation.

A browser must support HTML canvas to use this tool.

There is an online demo.

Importing data

Prepare a JSON file that looks like the following. The required fields are labels and imageURLs. The annotationURLs are for existing data and can be omitted. Place the JSON file inside the data/ directory.

{
  "labels": [
    "background",
    "skin",
    "hair",
    "dress",
    "glasses",
    "jacket",
    "skirt"
  ],
  "imageURLs": [
    "data/images/1.jpg",
    "data/images/2.jpg"
  ],
  "annotationURLs": [
    "data/annotations/1.png",
    "data/annotations/2.png"
  ]
}

Then edit main.js to point to this JSON file. Open a Web browser and visit index.html.

Known issues

Browser incompatibility

A segmentation result can greatly differ due to the difference in Javascript implementation across Web browsers. The difference stems from numerical precision of floating point numbers, and there is no easy way to produce the exact same result across browsers.

Python tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

import numpy as np
from PIL import Image

# Decode
encoded = np.array(Image.open('data/annotations/1.png'))
annotation = np.bitwise_or(np.bitwise_or(
    encoded[:, :, 0].astype(np.uint32),
    encoded[:, :, 1].astype(np.uint32) << 8),
    encoded[:, :, 2].astype(np.uint32) << 16)

print(np.unique(annotation))

# Encode
Image.fromarray(np.stack([
    np.bitwise_and(annotation, 255),
    np.bitwise_and(annotation >> 8, 255),
    np.bitwise_and(annotation >> 16, 255),
    ], axis=2).astype(np.uint8)).save('encoded.png')

JSON

Use JSON module.

import json

with open('data/example.json', 'r') as f:
    dataset = json.load(f)

Using dataURL

Do the following to convert between dataURL and NumPy format.

from PIL import Image
import base64
import io

# Encode
with io.BytesIO() as buffer:
    encoded.save(buffer, format='png')
    data_url = b'data:image/png;base64,' + base64.b64encode(buffer.getvalue())

# Decode
binary = base64.b64decode(data_url.replace(b'data:image/png;base64,', b''))
encoded = Image.open(io.BytesIO(binary))

Matlab tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

% Decode

X = imread('data/annotations/0.png');
annotation = X(:, :, 1);
annotation = bitor(annotation, bitshift(X(:, :, 2), 8));
annotation = bitor(annotation, bitshift(X(:, :, 3), 16));

% Encode

X = cat(3, bitand(annotation, 255), ...
           bitand(bitshift(annotation, -8), 255), ...
           bitand(bitshift(annotation, -16)), 255));
imwrite(uint8(X), 'data/annotations/0.png');

JSON

Use the matlab-json package.

Using dataURL

Get the byte encoding tools.

Do the following to convert between dataURL and Matlab format.

% Decode

dataURL = 'data:image/png;base64,...';
png_data = base64decode(strrep(dataURL, 'data:image/png;base64,', ''));
annotation = imdecode(png_data, 'png');

% Encode

png_data = imencode(annotation, 'png');
dataURL = ['data:image/png;base64,', base64encode(png_data)];

Citation

We appreciate if you cite the following article in an academic paper. The tool was originally developed for this work.

@article{tangseng2017looking,
Author        = {Pongsate Tangseng and Zhipeng Wu and Kota Yamaguchi},
Title         = {Looking at Outfit to Parse Clothing},
Eprint        = {1703.01386v1},
ArchivePrefix = {arXiv},
PrimaryClass  = {cs.CV},
Year          = {2017},
Month         = {Mar},
Url           = {http://arxiv.org/abs/1703.01386v1}
}
Comments
  • Tool crashes on loading Pre-Annotated Images

    Tool crashes on loading Pre-Annotated Images

    Hi @kyamagu ,

    Thank you for sharing this project.

    I'm trying to use custom annotation images and I've created the JSON file accordingly. The tool loads up the relevant classes color-coded over the image. But when I try to correct some segmentation the tool stops working and the following error is printed in the console.

    Browser Console Dump localhost-1505715475195.txt

    SampleJSON untitled )

    Sample Browser Screenshot untitled

    opened by anandcu3 15
  • cant compile paperdoll

    cant compile paperdoll

    Hi, sorry to write it here, i was not able to find a repo for paper doll. I can not compile the paper doll. it seems I have some problem with gco, if I comment it out everything else will work, then when I type in

    result = feature_calculator.apply(config, input_sample) I will get Invalid MEX-file '/BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64': /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64)

    Error in detect_fast>passmsg (line 128) [score0(:,:,k),Ix0(:,:,k),Iy0(:,:,k)] = shiftdt(child.score(:,:,k), child.w(1,k), child.w(2,k), child.w(3,k), child.w(4,k),child.startx(k),child.starty(k),Nx,Ny,child.step);

    Error in detect_fast (line 43) [msg,parts(k).Ix,parts(k).Iy,parts(k).Ik] = passmsg(parts(k),parts(par));

    Error in pose.estimate>process (line 38) box = detect_fast(im, model, detector_threshold);

    Error in pose.estimate (line 21) boxes{i} = process(model, samples{i}, scale, nms_threshold);

    Error in pose_calculator.apply (line 24) samples(i).(config.output) = pose.estimate(config.model, ...

    Error in feature_calculator.apply (line 25) sample = calculators{j}(config{j}, sample, varargin{:}, 'Encode', false);

    result =

     []
    

    can you help me with it please?

    opened by hosnasattar 11
  • Erase functionality

    Erase functionality

    I want to add an Erase button which erases any selection previously done. I used an button which when clicked calls a function very similar to the function responsible for Brush, which is:

    for (var y = -1 * brushSize; y <= brushSize; y++) {
        for (var x = -1 * brushSize; x <= brushSize; x++) {
          if (x * x + y * y > brushLimit) continue;
          var offset =
            4 *
            ((pos[1] + y) * this.layers.visualization.canvas.width + (pos[0] + x));
          offsets.push(offset);
          labels.push(label);
        }
      }
      annotator._updateAnnotation(offsets, labels);
    

    with one difference:

     labels.push(label);
    

    The last line inside the for loop I've hardcoded 0 being pushed into labels.

     labels.push(0);
    

    0 represents the background white color so on clicking the Erase button and selecting any area within a selection, it looks like stuff is being erased since the color of the label gets replaced by the background. But in truth this isn't actually erasing anything, just continues to add label with white background. So when I export it, the erased area expectedly still shows up.

    Any pointers on how to delete previously selected area?

    opened by Abdul-Ghani 10
  • How to code RGB and index information (8b)

    How to code RGB and index information (8b)

    Hi @kyamagu

    thank you for such a nice tool to annotate pictures. I'd like to ask if it is possible to do following:

    I already customized your tool and add 3 classes (Dress, Trousers, Jacket) as can be seen on this picture: 33 On the next picture below is annotated image from Pascal VOC 2012 for segmentation, containing RGB information + index for object and also the same for background. 11

    However, output from JS segmenter looks like this: 22

    If it is possible, could you please tell us on this example how to convert image from 32b to 8b and how to code the demanded information into image as it is in attached annotated VOC Pascal above. E.g. for background --> index 0 and RGB{0 0 0} for jacket --> index 3 and RGB{192 128 0}.

    Thank you in advance, much appreciated!

    opened by marticsvk 3
  • can't export annotation image with more than 2 images

    can't export annotation image with more than 2 images

    when I put more than 2 images in the imageURLs field in the json file, I can only view the first two images when I visit index.html. I can view all the other images by clicking "Next" though.

    The problem is: I can annotate the first two images and export the annotation pictures. But as for the other images, I am able to annotate the image, but not able to export it: There is no response when I click the "export" button.

    How can I solve this issue?

    p.s. The browser I use is Firefox(v42.0 Mac). index.html appeared to be blank with Chrome(v47.0 Mac) and Safari(v9.0).

    Thanks, john

    opened by johnzw 3
  • Some code using js-segment-annotator

    Some code using js-segment-annotator

    I am working on a code that uses this project, thought you might be interested.

    Repo: https://github.com/AKSHAYUBHAT/Vision Demo: http://vision.akshaybhat.com/ 3

    opened by ghost 3
  • Anyway way to debug when finding boundaries goes wrong?

    Anyway way to debug when finding boundaries goes wrong?

    I love your work here and greatly appreciate you sharing it. I'm not at all overly familiar with some of the concepts but wanted to use this in a proof of concept.

    I was using most of the default values and simply porting the code over to not be AMD modules. I have everything working except for the most important part - I've introduced a bug in the segmentation logic. What I end up with is almost perfect squares -

    image

    I've been combing over settings and options and trying to see what is different but nothing stands out. I know this isn't specific to your work on this library, but wanted to see if you had any tips for debugging this part of the code or if you had any tips for what could be going wrong?

    question 
    opened by plwalters 2
  • Annotated image are mostly zero pixel value

    Annotated image are mostly zero pixel value

    @kyamagu i used this tool for annotating my images for caffe-segnet,i first saw that all images are darker(you can hardly notice edges of annotated objects) but i thought on pixel level it might be different,but after checking in matlab,the whole image is almolt zero pixel.i would be happy if you could point out what i am doing wrong.thx

    opened by eliethesaiyan 2
  • Add a polygon-tool.

    Add a polygon-tool.

    • polygon tool
      • left click: span line between points
      • right click: erase drawn lines
    • keyboard shortcuts:
      • ctrl-key: switch between SLIC and polygon-tool
    • new buttons
    opened by NicolaiHarich 2
  • Image export

    Image export

    I cannot export the labeled image. I guess it was the incompatibility of the browser. Could you please suggest the right browser to use? Thanks in advance.

    SL

    question 
    opened by lsymuyu 1
  • Possible to update the annotation image after editing?

    Possible to update the annotation image after editing?

    I wanted to know if it is possible to have a save option which will update the annotation image in the data/annotations/ folder.

    If this is possible, where should I add this functionality?

    wontfix 
    opened by ghost 1
  • Polygon is difficult to close

    Polygon is difficult to close

    I find a bit difficult to close and complete a polygon. I checked the code but I don't understand where the relevant part is. How is the polygon closure checked?

    opened by FSet89 0
  • Brush tool used in the lateral border paints pixels on the other side of the image

    Brush tool used in the lateral border paints pixels on the other side of the image

    When using brush tool in the lateral border of one image, the area of the brush tool that stand out paints the pixels on the oposite border of the image.

    opened by polalbacar 0
  • The annotation masks disappear when clicked on Next button

    The annotation masks disappear when clicked on Next button

    I am able to export the annotated masks but as soon as I click on the "next" so that I can annotate the other images, the masks previously made are gone. We have to annotate the images again.

    It seems as if as soon as we hit "next", the app is removing the overlaid mask that has just been made and I am left with nothing. I have just cloned the library and started working.

    question 
    opened by nikhil-occipitaltech 5
  • Better annotation display

    Better annotation display

    Hi, Thanks for this awesome tool. May I propose an improvement in order to speed up the annotation process. I saw this in other tools and it was really efficient : the possibility to display raw images only in already annoted zones. I put below two examples of vegetation / soil annotation.

    Only vegetation toto

    Only soil: tata

    I checked your code but Im not so familiar with images overlay with javascript and I failed to display annotations with alpha = 1. Could you give me an advice ?

    enhancement question 
    opened by jerem-lab 3
  • The Upload image size is limited

    The Upload image size is limited

    HI, I just meet a problem when trying to upload images which are more than 50kb,these images showing in Firefox all are grey with nothing to see. Maybe the image compression is ok, beside that, any solutions ? Thank you for advance..

    wontfix 
    opened by wzx918 2
Releases(v2.0.1)
Owner
Kota Yamaguchi
Research scientist
Kota Yamaguchi
LSB Image Steganography Using Python

Steganography is the science that involves communicating secret data in an appropriate multimedia carrier, e.g., image, audio, and video files

Mahmut Can Gönül 2 Nov 04, 2021
Convert any image into greyscale ASCII art.

Image-to-ASCII Convert any image into greyscale ASCII art.

Ben Smith 12 Jan 15, 2022
Transfers a image file(.png) to an Excel file(.xlsx)

Transfers a image file(.png) to an Excel file(.xlsx)

Junu Kwon 7 Feb 11, 2022
Unique image & metadata generation using weighted layer collections.

nft-generator-py nft-generator-py is a python based NFT generator which programatically generates unique images using weighted layer files. The progra

Jonathan Becker 243 Dec 31, 2022
starfish is a Python library for processing images of image-based spatial transcriptomics.

starfish: scalable pipelines for image-based transcriptomics starfish is a Python library for processing images of image-based spatial transcriptomics

199 Dec 08, 2022
Imutils - A series of convenience functions to make basic image processing operations such as translation, rotation, resizing, skeletonization, and displaying Matplotlib images easier with OpenCV and Python.

imutils A series of convenience functions to make basic image processing functions such as translation, rotation, resizing, skeletonization, and displ

PyImageSearch 4.3k Jan 01, 2023
QR-Generator - An awesome QR Generator to create or customize your QR's

QR Generator An awesome QR Generator to create or customize your QR's! Table of

Tristán 1 Jan 28, 2022
vsketch is a Python generative art toolkit for plotters

Generative plotter art environment for Python

Antoine Beyeler 380 Dec 29, 2022
sK1 2.0 cross-platform vector graphics editor

sK1 2.0 sK1 2.0 is a cross-platform open source vector graphics editor similar to CorelDRAW, Adobe Illustrator, or Freehand. sK1 is oriented for prepr

sK1 Project 238 Dec 04, 2022
Fix datetime EXIF data in photos downloaded from Google Takeout

fix-google-takeout Warning Use at your own risk. Backup your photos. Overview Google takeout for photos

Mayank Mandava 20 Nov 05, 2022
SGTL - Spectral Graph Theory Library

SGTL - Spectral Graph Theory Library SGTL is a python library of spectral graph theory methods. The library is still very new and so there are many fe

Peter Macgregor 6 Oct 01, 2022
A simple image to text converter with GUI!

TEXTEMAGE! Textemage is a quick tool that extracts text from images, it is a Python based GUI program(also available in executable version). This is a

Akascape 5 Oct 26, 2022
Create a random fluent image based on multiple colors.

FluentGenerator Create a random fluent image based on multiple colors. Navigation Example Install Update Usage In Python console FluentGenerator Fluen

1 Feb 02, 2022
Anaglyph 3D Converter - A python script that adds a 3D anaglyph style effect to an image using the Pillow image processing package.

Anaglyph 3D Converter - A python script that adds a 3D anaglyph style effect to an image using the Pillow image processing package.

Kizdude 2 Jan 22, 2022
Python implementation of image filters (such as brightness, contrast, saturation, etc.)

PyPhotoshop Python implementation of image filters Use Python to adjust brightness and contrast, add blur, and detect edges! Follow along tutorial: ht

Kylie 87 Dec 15, 2022
Png2Jpg tool will help you convert from png image format to jpg images format.

PNG 2 JPG All codes assume running from root directory. Please update the sys path at the beginning of the codes before running. Over View Png2Jpg too

Nguyễn Trường Lâu 2 Dec 27, 2021
A little Python tool to convert a TrueType (ttf/otf) font into a PNG for use in demos.

font2png A little Python tool to convert a TrueType (ttf/otf) font into a PNG for use in demos. To use from command line it expects python3 to be at /

Rich Elmes 3 Dec 22, 2021
Convert the SVG code to PNG and replace the line by a call to the image in markdown

Convert the SVG code to PNG and replace the line by a call to the image in markdown

6 Sep 06, 2022
An executor that performs standard pre-processing and normalization on images.

An executor that performs standard pre-processing and normalization on images.

Jina AI 6 Jun 30, 2022