Encode a sentence (with UNK)

Background

With a vocabulary in hand, a sentence becomes a list of token IDs. Any token not in the vocabulary maps to a special unknown index, so the model can still process out-of-vocabulary words.

Problem statement

Implement encode(tokens, vocab, unk_index) returning the list of IDs, using unk_index for tokens missing from vocab.

Input

• tokens — list of string tokens.
• vocab — dict {token: index}.
• unk_index — int, the fallback index for unknown tokens.

Output

Returns a list of int IDs, one per token.

Examples

Example 1

Input:  tokens = ["i", "cat", "i"], vocab = {"i": 0, "cat": 1}, unk_index = 99
Output: [0, 1, 0]

Example 2 — out-of-vocabulary

Input:  tokens = ["i", "dog"], vocab = {"i": 0, "cat": 1}, unk_index = 99
Output: [0, 99]

Explanation: "dog" isn't in the vocab, so it maps to unk_index.

Constraints

• Look up each token; use unk_index when it's missing.
• Return a list of ints, same length as tokens.

Notes

• This is the bridge from text to the integer sequence an RNN (or transformer) consumes; the embedding layer then turns each ID into a vector.