Base Architectures

class multivae.models.nn.BaseJointEncoder

This is a base class for Joint Encoders neural networks.

forward(x)

This function must be implemented in a child class. It takes the input data and returns an instance of ModelOutput. If you decide to provide your own joint encoder network, you must make sure your model inherit from this class by setting and then defining your forward function as below.

>>> from multivae.models.nn import BaseEncoder
>>> from pythae.models.base.base_utils import ModelOutput
...
>>> class My_Joint_Encoder(BaseEncoder):
...
...     def __init__(self):
...         BaseEncoder.__init__(self)
...         # your code
...         self.latent_dim = ...
...
...     def forward(self, x: dict):
...         # x is a dict with a tensor for each modality
...         # your code
...         output = ModelOutput(
...             embedding=embedding,
...             log_covariance=log_var # for VAE based models
...         )
...         return output

Parameters:

x (dict) – Multimodal input to encode : a dictionary that contains modalities’ names as keys and modalities’ data as values.

Returns:

The output of the encoder

Return type:

output (ModelOutput)

class multivae.models.nn.BaseConditionalDecoder

This is a base class for Conditional Decoders architectures.

forward(z, cond_mods)

This function must be implemented in a child class. It takes the latent variable z and conditioning modality and returns an instance of ModelOutput with the reconstruction. If you decide to provide your own decoder network, you must make sure your model inherit from this class by setting and then defining your forward function as below.

>>> from pythae.models.nn import BaseConditionalDecoder
>>> from pythae.models.base.base_utils import ModelOutput
...
>>> class My_Conditional_Decoder(BaseConditionalDecoder):
...
...     def __init__(self):
...         BaseConditionalDecoder.__init__(self)
...         # your code
...         self.latent_dim = ...
...
...     def forward(self, z, cond_mods):
...         # your code
...         output = ModelOutput(
...             reconstruction= ...
...         )
...         return output

Parameters:

• z (torch.Tensor) – Latent variable

• cond_mods (Dic[str, torch.Tensor]) – Conditioning data.

Returns:

The output of the decoder.

Return type:

output (ModelOutput)

class multivae.models.nn.BaseMultilatentEncoder

This is a base class for for encoders with multiple latent spaces.

forward(x)

This function must be implemented in a child class. It takes the input tensor x and returns an instance of ModelOutput with the parameters for the shared latent space and the modality-specific latent space. If you decide to provide your own encoder network in a model that uses multiple latent spaces, you must make sure your model inherits from this class by setting and then defining your forward function as below.

>>> from multivae.models.nn import BaseMultilatentEncoder
>>> from pythae.models.base.base_utils import ModelOutput
...
>>> class My_Encoder(BaseMultilatentEncoder):
...
...     def __init__(self):
...         BaseMultilatentEncoder.__init__(self)
...         # your code
...         self.latent_dim = ...
...         self.style_dim = ...
...
...     def forward(self, x):
...         # your code
...         output = ModelOutput(
...             embedding= ..., # shared latent space
                log_covariance=...,
                style_embedding=..., # modality-specific latent space
                style_log_covariance=...,
...         )
...         return output

Parameters:

x (torch.Tensor) – Input data

Returns:

The output of the encoder.

Return type:

output (ModelOutput)