Understanding Transformers Part 11: How Decoding Begins

In the previous article we wrapped up the encoder part, In this article, we will start building the second part of the transformer: the decoder. Just like the encoder, the decoder also begins with word embeddings. However, this time the embeddings are created for the output vocabulary, which consists of Spanish words such as: ir vamos y token Starting the Decoding Process To begin decoding, we use the token as the input. This is a common way to initialize the decoding process for an encoded sentence. In some cases, people use a (Start of Sentence) token instead.