Building a simple Generative Adversarial Network (GAN) using TensorFlow

Sep 26, 2024 06:28 PM - 4 months ago 156730

Introduction

Generative Adversarial Networks aliases GANs are 1 of the astir progressive areas successful heavy learning investigation and improvement owed to their unthinkable expertise to make synthetic results. In this blog, we will build retired the basal intuition of GANs done a actual example. This station is surgery down successful pursuing way:

  • Basic thought and intuition down workings of Generative Adversarial Networks
  • Implementing a GAN-based exemplary that generates information from a elemental distribution
  • Visualizing and analyzing different aspects of the GAN to amended understand what’s happening down the scenes.

The codification for this blog tin beryllium recovered here.

Generative Adversarial Networks

The basal thought down GANs is really very simple. At its core, a GAN includes 2 agents pinch competing objectives that activity done opposing goals. This comparatively elemental setup results successful some of the agent’s coming up pinch progressively analyzable ways to deceive each other. This benignant of business tin beryllium modeled successful Game Theory arsenic a minimax game.

Let’s return a theoretical illustration of the process of money counterfeiting. In this process, we tin ideate 2 types agents: a criminal and cop. Let america look into their competing objectives:

  • Criminal’s Objective: The main nonsubjective of the criminal is to travel up pinch analyzable ways of counterfeiting money specified that the Cop cannot separate betwixt counterfeited money and existent money.
  • Cop’s Objective: The main nonsubjective of the bull is to travel up pinch analyzable ways truthful arsenic to separate betwixt counterfeited money and existent money.

As this process progresses the cop develops much and much blase exertion to observe money counterfeiting and criminal develops much and much blase exertion to counterfeit money. This is the ground of what is called an Adversarial Process

Generative Adversarial Networks return advantage of Adversarial Processes to train 2 Neural Networks who compete pinch each different until a desirable equilibrium is reached. In this case, we person a Generator Network G(Z) which takes input random sound and tries to make information very adjacent to the dataset we have. The different web is called the Discriminator Network D(X) which takes input generated information and tries to discriminate betwixt generated information and existent data. This web astatine its halfway implements a binary classification and outputs the probability that the input information really comes from the existent dataset (as opposed to the synthetic, aliases clone data).

In the general consciousness the nonsubjective usability of this full process tin beryllium written as:

GAN's nonsubjective function

The accustomed desirable equilibrium constituent for the supra defined GANs is that the Generator should exemplary the existent information and Discriminator should output the probability of 0.5 arsenic the generated information is aforesaid arsenic the existent information – that is, it is not judge if the caller information coming from the generator is existent aliases clone pinch adjacent probability.

You mightiness beryllium wondering why specified a analyzable learning process is moreover required? What are the advantages of learning specified a model? Well, the intuition down this and each the generative approaches travel a celebrated quote from Richard Feynman:

What I cannot create, I do not understand.

This is applicable because if we are capable to make existent information distribution from a exemplary past it intends that we cognize everything that this to cognize astir that model. A batch of clip these existent distributions see millions of images and we tin make them utilizing a exemplary that has thousands of parameters past these parameters seizure the principle of the fixed images.

GANs person galore different real-life short-term applications besides which we will talk successful a later section.

Implementing GANs

In this section, we will make a very elemental information distribution and effort to study a Generator usability that generates information from this distribution utilizing GANs exemplary described above. This conception is broadly divided into 3 parts. Firstly we will constitute a basal usability to make a quadratic distribution (the existent information distribution). Secondly, we constitute codification for Generator and Discriminator networks. Then we will usage the information and the networks to constitute the codification for training these 2 networks successful an adversarial way.

The nonsubjective of this implementation is to study a caller usability that tin make information from the aforesaid distribution arsenic the training data. The anticipation from the training is that our Generator web should commencement producing information which follows the quadratic distribution. This is explained and demonstrated much successful the adjacent section. Although we are starting pinch very elemental information distribution, this attack tin beryllium easy extended to make information from the overmuch much analyzable dataset. Few illustration GANs person successfully generated images of handwritten digits, faces of celebrities, animals, etc.

Generating Training Data

We instrumentality our existent dataset by generating random samples utilizing numpy room and past generating the 2nd coordinate utilizing immoderate benignant of function. For the intent of this demo, we person kept the usability arsenic a quadratic usability for simplicity. You tin play pinch this codification to make a dataset pinch much dimensions and/or much analyzable narration betwixt its features specified arsenic higher grade polynomial, sine, cosine, etc.

import numpy as np def get_y(x): return 10 + x*x def sample_data(n=10000, scale=100): information = [] x = scale*(np.random.random_sample((n,))-0.5) for one in range(n): yi = get_y(x[i]) data.append([x[i], yi]) return np.array(data)

The generated information is very elemental and tin beryllium plotted arsenic seen here:

Training data

Generator and Discriminator Networks Implementation

We will now instrumentality the Generator and Discriminator networks utilizing tensorflow layers. We instrumentality the Generator web utilizing the pursuing function:

def generator(Z,hsize=[16, 16],reuse=False): with tf.variable_scope("GAN/Generator",reuse=reuse): h1 = tf.layers.dense(Z,hsize[0],activation=tf.nn.leaky_relu) h2 = tf.layers.dense(h1,hsize[1],activation=tf.nn.leaky_relu) retired = tf.layers.dense(h2,2) return out

This usability takes successful the placeholder for random samples (Z), an array hsize for the number of units successful the 2 hidden layers and a reuse adaptable which is utilized for reusing the aforesaid layers. Using these inputs it creates a afloat connected neural web of 2 hidden layers pinch fixed number of nodes. The output of this usability is simply a 2-dimensional vector which corresponds to the dimensions of the existent dataset that we are trying to learn. The supra usability tin beryllium easy modified to see much hidden layers, different types of layers, different activation and different output mappings.

We instrumentality the Discriminator web utilizing the pursuing function:

def discriminator(X,hsize=[16, 16],reuse=False): with tf.variable_scope("GAN/Discriminator",reuse=reuse): h1 = tf.layers.dense(X,hsize[0],activation=tf.nn.leaky_relu) h2 = tf.layers.dense(h1,hsize[1],activation=tf.nn.leaky_relu) h3 = tf.layers.dense(h2,2) retired = tf.layers.dense(h3,1) return out, h3

This usability takes input placeholder for the samples from the vector abstraction of existent dataset. The samples tin beryllium some existent samples and samples generated from the Generator network. Similar to the Generator web supra it besides takes input hsize and reuse. We usage 3 hidden layers for the Discriminator retired of which first 2 layers size we return input. We hole the size of the 3rd hidden furniture to 2 truthful that we tin visualize the transformed characteristic abstraction successful a 2D level arsenic explained successful the later section. The output of this usability is simply a logit prediction for the fixed X and the output of the past furniture which is the characteristic translator learned by Discriminator for X. The logit usability is the inverse of the sigmoid usability which is utilized to correspond the logarithm of the likelihood (ratio of the probability of adaptable being 1 to that of it being 0).

Adversarial Training

For the intent of training we specify the pursuing placeholders X and Z for existent samples and random sound samples respectively:

X = tf.placeholder(tf.float32,[None,2]) Z = tf.placeholder(tf.float32,[None,2])

We besides request to create the chart for generating samples from Generator web and feeding existent and generated samples to the Discriminator network. This is done by utilizing the functions and placeholders defined above:

G_sample = generator(Z) r_logits, r_rep = discriminator(X) f_logits, g_rep = discriminator(G_sample,reuse=True)

Using the logits for generated information and existent information we specify the nonaccomplishment functions for the Generator and Discriminator networks arsenic follows:

disc_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=r_logits,labels=tf.ones_like(r_logits)) + tf.nn.sigmoid_cross_entropy_with_logits(logits=f_logits,labels=tf.zeros_like(f_logits))) gen_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=f_logits,labels=tf.ones_like(f_logits)))

These losses are sigmoid transverse entropy based losses utilizing the equations we defined above. This is simply a commonly utilized nonaccomplishment usability for alleged discrete classification. It takes arsenic input the logit (which is fixed by our discriminator network) and existent labels for each sample. It past calculates the correction for each sample. We are utilizing the optimized type of this arsenic implemented by TensorFlow which is much unchangeable past straight taking calculating transverse entropy. For much details, you tin cheque retired the applicable TensorFlow API here.

Next, we specify the optimizers for the 2 networks utilizing the nonaccomplishment functions defined supra and scope of the layers defined successful the generator and discriminator functions. We usage RMSProp Optimizer for some the networks pinch the learning complaint arsenic 0.001. Using the scope we fetch the weights/variables for the fixed web only.

gen_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,scope="GAN/Generator") disc_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,scope="GAN/Discriminator") gen_step = tf.train.RMSPropOptimizer(learning_rate=0.001).minimize(gen_loss,var_list = gen_vars) disc_step = tf.train.RMSPropOptimizer(learning_rate=0.001).minimize(disc_loss,var_list = disc_vars)

We past train some the networks successful an alternating measurement for the required number of steps:

for one in range(100001): X_batch = sample_data(n=batch_size) Z_batch = sample_Z(batch_size, 2) _, dloss = sess.run([disc_step, disc_loss], feed_dict={X: X_batch, Z: Z_batch}) _, gloss = sess.run([gen_step, gen_loss], feed_dict={Z: Z_batch}) print "Iterations: %d\t Discriminator loss: %.4f\t Generator loss: %.4f"%(i,dloss,gloss)

The supra codification tin beryllium modified to see much analyzable training procedures specified arsenic moving aggregate steps of the discriminator and/or generator update, fetching the features of the existent and generated samples and plotting the generated samples. Please mention to the codification repository for specified modifications.

Analyzing GANs

Visualizing the Training losses

To amended understand what is happening successful this process, we tin crippled the training losses aft each 10 iterations. From the crippled beneath we tin spot really changes successful nonaccomplishment alteration gradually and that nonaccomplishment becomes almost changeless towards the extremity of training. This negligible alteration successful the nonaccomplishment of some Discriminator
and Generator indicates equilibrium.

Training Loss

Visualizing Samples during Training

We tin besides crippled the existent and generated samples aft each 1000 iterations of training. These land visualize really the Generator web starts pinch a random first mapping betwixt the input and dataset vector abstraction and past it gradually progresses to lucifer the existent dataset samples. As you tin see, the “fake” sample starts looking much and much for illustration the “real” information distribution.

samples_training

Visualizing the Generator Update

In this section, we visualize the effect of updating the Generator web weights wrong the adversarial trainingprocess. We do this by plotting the activations of the past hidden furniture of Discriminator network. We chose the past hidden furniture to beryllium of size 2 truthful that it would beryllium easy to crippled without requiring dimensionality simplification (i.e. the translator of the input sample to a different vector space). We are willing successful visualizing the characteristic translator usability learned by the Discriminator network. This usability is what our web learns truthful that the existent and clone information are separable.

We crippled the characteristic toggle shape of the existent and generated samples arsenic learned by the Discriminator network’s past furniture earlier and aft we update the weights of the Generator network. We besides crippled the centroids of the points that we get aft the characteristic translator of the input samples. Finally, we cipher the centroids of the points separately for some existent and clone data, earlier and aft the generator update. From the land we tin infer pursuing things:

  • As expected location is nary alteration successful the transformed features of the existent information samples. From the plots, we tin spot they wholly coincide.
  • From the centroids, we tin spot that centroid of the features of generated information samples almost ever moves towards the centroid of the features of existent information samples.
  • We tin besides spot that arsenic the iterations summation the transformed features of existent samples get much and much mixed pinch transformed features of generated samples. This is besides expected because astatine the extremity of training the Discriminator web should not beryllium capable to separate betwixt existent and generated samples. Hence astatine the extremity of training transformed features for some the samples should coincide.

feature_transform feature_transform_centroid

Conclusion and Future Work

We person implemented a proof-of-concept GAN exemplary for generating information from a very elemental information distribution. As an workout for the funny reader, we urge modifying the supra codification to do the following:

  • Visualize the earlier and aft of the Discriminator update.
  • Change the activation functions of the layers and spot the quality successful training and generated samples.
  • Add much layers and different types of layers and spot the effect connected the training clip and the stableness of the training.
  • Modify the codification for generating information to see information from 2 different curves
  • Modify the supra codification to activity pinch much analyzable information specified arsenic MNIST, CIFAR-10, etc.

In early work, we will talk limitations of the GANs and the modifications that are required to lick them.

More