TextClassifierPreprocessor

• 원본 링크 : https://keras.io/api/keras_nlp/base_classes/text_classifier_preprocessor/
• 최종 확인 : 2024-11-26

source

TextClassifierPreprocessor class

keras_nlp.models.TextClassifierPreprocessor(
    tokenizer, sequence_length=512, truncate="round_robin", **kwargs
)

Base class for text classification preprocessing layers.

TextClassifierPreprocessor tasks wrap a keras_hub.tokenizer.Tokenizer to create a preprocessing layer for text classification tasks. It is intended to be paired with a keras_hub.models.TextClassifier task.

All TextClassifierPreprocessor take inputs three ordered inputs, x, y, and sample_weight. x, the first input, should always be included. It can be a single string, a batch of strings, or a tuple of batches of string segments that should be combined into a single sequence. See examples below. y and sample_weight are optional inputs that will be passed through unaltered. Usually, y will be the classification label, and sample_weight will not be provided.

The layer will output either x, an (x, y) tuple if labels were provided, or an (x, y, sample_weight) tuple if labels and sample weight were provided. x will be a dictionary with tokenized input, the exact contents of the dictionary will depend on the model being used.

All TextClassifierPreprocessor tasks include a from_preset() constructor which can be used to load a pre-trained config and vocabularies. You can call the from_preset() constructor directly on this base class, in which case the correct class for you model will be automatically instantiated.

Examples.

preprocessor = keras_hub.models.TextClassifierPreprocessor.from_preset(
    "bert_base_en_uncased",
    sequence_length=256, # Optional.
)
# Tokenize and pad/truncate a single sentence.
x = "The quick brown fox jumped."
x = preprocessor(x)
# Tokenize and pad/truncate a labeled sentence.
x, y = "The quick brown fox jumped.", 1
x, y = preprocessor(x, y)
# Tokenize and pad/truncate a batch of labeled sentences.
x, y = ["The quick brown fox jumped.", "Call me Ishmael."], [1, 0]
x, y = preprocessor(x, y)
# Tokenize and combine a batch of labeled sentence pairs.
first = ["The quick brown fox jumped.", "Call me Ishmael."]
second = ["The fox tripped.", "Oh look, a whale."]
labels = [1, 0]
x, y = (first, second), labels
x, y = preprocessor(x, y)
# Use a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset).
ds = tf.data.Dataset.from_tensor_slices(((first, second), labels))
ds = ds.map(preprocessor, num_parallel_calls=tf.data.AUTOTUNE)

source

from_preset method

TextClassifierPreprocessor.from_preset(
    preset, config_file="preprocessor.json", **kwargs
)

Instantiate a keras_hub.models.Preprocessor from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:

1. a built-in preset identifier like 'bert_base_en'
2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
3. a Hugging Face handle like 'hf://user/bert_base_en'
4. a path to a local preset directory like './bert_base_en'

For any Preprocessor subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

As there are usually multiple preprocessing classes for a given model, this method should be called on a specific subclass like keras_hub.models.BertTextClassifierPreprocessor.from_preset().

Arguments

• preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.

Examples

# Load a preprocessor for Gemma generation.
preprocessor = keras_hub.models.GemmaCausalLMPreprocessor.from_preset(
    "gemma_2b_en",
)
# Load a preprocessor for Bert classification.
preprocessor = keras_hub.models.BertTextClassifierPreprocessor.from_preset(
    "bert_base_en",
)

source

save_to_preset method

TextClassifierPreprocessor.save_to_preset(preset_dir)

Save preprocessor to a preset directory.

Arguments

• preset_dir: The path to the local model preset directory.

tokenizer property

keras_nlp.models.TextClassifierPreprocessor.tokenizer

The tokenizer used to tokenize strings.