SentencePieceTokenizer

Original Link : https://keras.io/api/keras_nlp/tokenizers/sentence_piece_tokenizer/
Last Checked at : 2024-11-26

`SentencePieceTokenizer` class

keras_nlp.tokenizers.SentencePieceTokenizer(
    proto=None,
    sequence_length=None,
    dtype="int32",
    add_bos=False,
    add_eos=False,
    **kwargs
)

A SentencePiece tokenizer layer.

This layer provides an implementation of SentencePiece tokenization as described in the SentencePiece paper and the SentencePiece package. The tokenization will run entirely within the Tensorflow graph, and can be saved inside a keras.Model.

By default, the layer will output a tf.RaggedTensor where the last dimension of the output is ragged after whitespace splitting and sub-word tokenizing. If sequence_length is set, the layer will output a dense tf.Tensor where all inputs have been padded or truncated to sequence_length. The output dtype can be controlled via the dtype argument, which should be either an integer or string type.

Arguments

proto: Either a string path to a SentencePiece proto file, or a bytes object with a serialized SentencePiece proto. See the SentencePiece repository for more details on the format.
sequence_length: If set, the output will be converted to a dense tensor and padded/trimmed so all outputs are of sequence_length.
add_bos: Add beginning of sentence token to the result.
add_eos: Add end of sentence token to the result. Token is always truncated if output is longer than specified sequence_length.

References

Kudo and Richardson, 2018

Examples

From bytes.

def train_sentence_piece_bytes(ds, size):
    bytes_io = io.BytesIO()
    sentencepiece.SentencePieceTrainer.train(
        sentence_iterator=ds.as_numpy_iterator(),
        model_writer=bytes_io,
        vocab_size=size,
    )
    return bytes_io.getvalue()
# Train a sentencepiece proto.
ds = tf.data.Dataset.from_tensor_slices(["the quick brown fox."])
proto = train_sentence_piece_bytes(ds, 20)
# Tokenize inputs.
tokenizer = keras_hub.tokenizers.SentencePieceTokenizer(proto=proto)
ds = ds.map(tokenizer)

From a file.

def train_sentence_piece_file(ds, path, size):
    with open(path, "wb") as model_file:
        sentencepiece.SentencePieceTrainer.train(
            sentence_iterator=ds.as_numpy_iterator(),
            model_writer=model_file,
            vocab_size=size,
        )
# Train a sentencepiece proto.
ds = tf.data.Dataset.from_tensor_slices(["the quick brown fox."])
proto = train_sentence_piece_file(ds, "model.spm", 20)
# Tokenize inputs.
tokenizer = keras_hub.tokenizers.SentencePieceTokenizer(proto="model.spm")
ds = ds.map(tokenizer)

source

`tokenize` method

SentencePieceTokenizer.tokenize(inputs)

Transform input tensors of strings into output tokens.

Arguments

inputs: Input tensor, or dict/list/tuple of input tensors.
*args: Additional positional arguments.
**kwargs: Additional keyword arguments.

source

`detokenize` method

SentencePieceTokenizer.detokenize(inputs)

Transform tokens back into strings.

Arguments

inputs: Input tensor, or dict/list/tuple of input tensors.
*args: Additional positional arguments.
**kwargs: Additional keyword arguments.

source

`get_vocabulary` method

SentencePieceTokenizer.get_vocabulary()

Get the tokenizer vocabulary.

source

`vocabulary_size` method

SentencePieceTokenizer.vocabulary_size()

Get the integer size of the tokenizer vocabulary.

source

`token_to_id` method

SentencePieceTokenizer.token_to_id(token)

Convert a string token to an integer id.

source

`id_to_token` method

SentencePieceTokenizer.id_to_token(id)

Convert an integer id to a string token.

SentencePieceTokenizer