mwptoolkit.module.Embedder.position_embedder

class mwptoolkit.module.Embedder.position_embedder.DisPositionalEncoding(embedding_size, max_len)

Bases: Module

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(dis_graph, category_num)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class mwptoolkit.module.Embedder.position_embedder.EPTPositionalEncoding(embedding_dim)

Bases: Module

Positional encoding that extends trigonometric embedding proposed in ‘Attention is all you need’

Instantiate positional encoding instance.

Parameters

embedding_dim (int) – Dimension of embedding vector

_forward(index_or_range, ignored_index=-1) → Tensor

Compute positional encoding

\[P_{t, p} = c_p * \cos(a_p * t + b_p) + d_p * \sin(a_p * t + b_p).\]

Parameters

• index_or_range (Union[torch.Tensor,int,range]) – Value that represents positional encodings to be built. - A Tensor value indicates indices itself. - A integer value indicates indices from 0 to the value - A range value indicates indices within the range.

• ignored_index (int) – The index to be ignored. PAD_ID by default.

Return type

torch.Tensor

Returns

Positional encoding of given value. - If torch.Tensor of shape [, L] is given, this will have shape [, L, E] if L is not 1, otherwise [*, E]. - If integer or range is given, this will have shape [T, E], where T is the length of range.

before_trigonometric(indices: Tensor) → Tensor

Compute a_p * t + b_p for each index t. :param torch.Tensor indices: A Long tensor to compute indices. :rtype: torch.Tensor :return: Tensor whose values are a_p * t + b_p for each (t, p) entry.

property device: device

Get the device where weights are currently put. :rtype: torch.device :return: Device instance

embedding_dim

Dimension of embedding vector

forward(index_or_range, ignored_index=-1) → Tensor

Compute positional encoding. If this encoding is not learnable, the result cannot have any gradient vector.

\[P_{t, p} = c_p * \cos(a_p * t + b_p) + d_p * \sin(a_p * t + b_p).\]

Parameters

• index_or_range (Union[torch.Tensor,int,range]) – Value that represents positional encodings to be built. - A Tensor value indicates indices itself. - A integer value indicates indices from 0 to the value - A range value indicates indices within the range.

• ignored_index (int) – The index to be ignored. PAD_ID by default.

Return type

torch.Tensor

Returns

Positional encoding of given value. - If torch.Tensor of shape [, L] is given, this will have shape [, L, E] if L is not 1, otherwise [*, E]. - If integer or range is given, this will have shape [T, E], where T is the length of range.

training: bool

class mwptoolkit.module.Embedder.position_embedder.PositionEmbedder(embedding_size, max_length=512)

Bases: Module

This module produces sinusoidal positional embeddings of any length.

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(input_seq, offset=0)

Parameters

input_seq (torch.Tensor) – input sequence, shape [batch_size, sequence_length].

Returns

position embedding, shape [batch_size, sequence_length, embedding_size].

Return type

torch.Tensor

get_embedding(max_length, embedding_size)

Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of “Attention Is All You Need”.

training: bool

class mwptoolkit.module.Embedder.position_embedder.PositionEmbedder_x(embedding_size, max_len=1024)

Bases: Module

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(input_embedding)

Parameters

input_embedding (torch.Tensor) – shape [batch_size, sequence_length, embedding_size].

training: bool

class mwptoolkit.module.Embedder.position_embedder.PositionalEncoding(pos_size, dim)

Bases: Module

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(input)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool