Autoencoder

What is Autoencoder#

An autoencoder is a type of algorithm with the primary
purpose of learning an “informative” representation of the data that
can be used for different applications by learning to reconstruct a set
of input observations well enough.

https://arxiv.org/abs/2003.05991 ↗

There are three main components in the autoencoders: encoder. latent feature representation and a decoder.

The encoder and decoder are simply functions, while with the name latent feature representation, one typically intends a tensor of real numbers

https://arxiv.org/pdf/2201.03898 ↗

If Decoder and Encoder are linear model, the autoencoder will achieve same latent presentation of PCA.

to prove:http://toe.lt/1a ↗.

https://arxiv.org/abs/1804.10253 ↗

The mission of autoencoder is to learn the functions $\mathit{A}:\mathbb{R}^{\mathit{n}}\to\mathbb{R}^{\mathit{p}}$ (encoder) and $\mathit{B}:\mathbb{R}^{\mathit{p}}\to\mathbb{R}^{\mathit{n}}$ (decoder) that satisfy

$\mathit{M}$
where $\mathit{E}$ is the expectation over the distribution of $\mathit{x}$ , and $\Delta$ is the reconstruction loss function, which measures the distance between the output of the decoder and the intput. The latter is usually set to be the $\ell_{2}$ -norm.

Easiest way to realize a autoencoder#

class Encoder(nn.Module):
    def __init__(self, input_dim, feature_dim):
        super(Encoder, self).__init__()
        self.fc = nn.Linear(input_dim, feature_dim)

    def forward(self, x):
        x = torch.relu(self.fc(x))
        return x

class Decoder(nn.Module):
    def __init__(self, feature_dim, output_dim):
        super(Decoder, self).__init__()
        self.fc = nn.Linear(feature_dim, output_dim)

    def forward(self, x):
        x = torch.sigmoid(self.fc(x))
        return x

class Autoencoder(nn.Module):
    def __init__(self, input_dim, feature_dim):
        super(Autoencoder, self).__init__()
        self.encoder = Encoder(input_dim, feature_dim)
        self.decoder = Decoder(feature_dim, input_dim)

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded