Estimators in Scikit-LLM: A KDnuggets Cheat Sheet

There are two ways you can go about integrating LLMs into your traditional machine learning workflows. One is using scikit-learn, with pipelines and cross-validation and a metrics report at the end. The other is a script full of loops over API calls, string parsing, and a try/except wrapped around a response that occasionally came back as prose instead of a label. Both perform classification, but only the first provided an approach worth reusing.

Scikit-LLM closes that gap by wrapping language models in the scikit-learn estimator API you may already be using everywhere else. Every model has fit and either predict or transform, so it drops into a Pipeline or a cross-validation loop natively. What actually differs is fit: it usually just records the label set, because the work happens at predict time, one API call per sample. You can think and plan in tokens, which is the idea this new cheat sheet is built around.

The one you will likely end up using the most is ZeroShotGPTClassifier, and it took me some time to figure it out at first. Calling fit(None, [...]) with nothing but candidate labels feels wrong the first few times, until you internalize that the labels are the task specification. Vague labels give vague results, so it's best to treat them as descriptions. When zero-shot isn't enough, DynamicFewShotGPTClassifier is the one to default to over plain few-shot, since it cleverly retrieves the closest examples per class per sample instead of using the entire training set into every prompt.

There are other gems here as well. GPTVectorizer turns text of any length into a fixed-width vector, so the LLM becomes step one of a pipeline and everything after it is just plain old scikit-learn — so you can run a logistic regression on embeddings. And GPTTranslator is a transformer, so it can be positioned ahead of a classifier that only ever saw English in training, with no need for retraining on a multilingual corpus.

The one caveat I would stress is the actual token cost involved. A cross_val_score with cv=3 is three times the API calls, and that multiplies with a grid search you were previously running without a second thought. The habits that didn't cost us anything in scikit-learn are not free here.

Keeping that in mind, scikit-LLM is a fantastic tool to add to your AI engineering toolkit, especially if you are working with scikit-learn often. It's also a great approach to getting your hands dirty without straying too far from your comfort zone.

Check out the new cheat sheet right now to keep the scikit-LLM essentials handy while you get started with this great sidekick library, or for reference as a regular user.