Adain : A Machine Learning Model for Style Transfer

This is an introduction to「Adain」, a machine learning model that can be used with ailia SDK. You can easily use this model to create AI applications using ailia SDK as well as many other ready-to-use ailia MODELS.

Overview

Adain is a machine learning model that transforms the style of an image.

Arbitrary Style Transfer in Real-time with Adaptive Instance NormalizationGatys et al. recently introduced a neural algorithm that renders a content image in the style of another image…arxiv.org

Conventional style transfer requires learning for each new style, but Adain is able to perform style transformation for any image without relearning.

Adain takes an base image and a style reference image as input, and transforms the style of the base image.

Base input image（Source：https://github.com/naoto0804/pytorch-AdaIN/tree/master/input）

Style reference image（Source：https://github.com/naoto0804/pytorch-AdaIN/tree/master/input）

Result image（Source：https://github.com/naoto0804/pytorch-AdaIN/tree/master/input）

Architecture

Adain stands for Adaptive Instance Normalization, and the architecture of Adain is as follows.

In the study of style transfer, it is known that the style of an image can be transformed by rewriting the mean and variance of VGG’s feature maps.

Adain applies VGG to the base and style reference images to obtain the feature maps, and then replaces the mean and variance of the feature maps up to Relu4 of the base image with the mean and variance of the style reference image. The final image is then generated by passing it through a decoder.

Adaptive Instance Normalization is an improved version of Instance Normalization that calculates the mean and variance using the same operations as regular Instance Normalization, and then replaces them with the style mean and variance.

For the actual network definition, please refer to the repository below.

naoto0804/pytorch-AdaINThis is an unofficial pytorch implementation of a paper, Arbitrary Style Transfer in Real-time with Adaptive Instance…github.com

ReflectionPad2d is used in the decoder, and the output ONNX is rather complex.

MSCOCO and Wikiart were used for training.

Usage

You can refer to the following sample to run Adain with ailia SDK.

ailia-ai/ailia-models(These images from https://github.com/naoto0804/pytorch-AdaIN/tree/master/input) Ailia input shape: (1, 3, 512, 512)…github.com

Use the command below to apply Adain to a style image.

$ python3 adain.py

Use the command below to apply Adain to the webcam video stream.

$ python3 adain.py -v 0

ailia Inc. has developed ailia SDK, which enables cross-platform, GPU-based rapid inference.

ailia Inc. provides a wide range of services from consulting and model creation, to the development of AI-based applications and SDKs. Feel free to contact us for any inquiry.