Workshop: text and language models

Cultural Data Analysis · Week 4: Cultural Data Forms: Text

This week has the text workshop with transformer models, and a group exercise on language models and their APIs, with a choice of two notebooks.

First: save your own copy. Unless the notebook is already in your own Google Drive, Colab does not keep your changes. Go to File → Save a copy in Drive and work in the copy that opens in a new tab. You find it later in Google Drive, in Colab Notebooks.

1. Text workshop: transformers

Sentiment analysis of social media with Hugging Face: datasets, tokenisation (characters, words, sub-words), a baseline, fine-tuning a transformer model, and a second case study (do two sentences mean the same thing?). Then the steps of the lecture's traditional NLP again, with language models and without any training: sentiment, topics, entities and a chat model.

To submit: run the whole notebook and complete every cell marked #Complete cell. Sections and cells marked optional you can skip.

Open in ColabWeek4_Workshop.ipynb

Before you run anything:

In the top menu, go to Runtime → Change runtime type and choose Python 3 with hardware accelerator T4 GPU.
When asked, allow the notebook access to Google Drive: choose the same Google account you use for Colab and tick all the permissions. The notebook reads the models from the shared course cache (the CDA-CACHE shortcut you set up following the instructions on Canvas).
Run the first code cell. It takes a while.

Hand it in

The last cell of the notebook makes an HTML file of your notebook and downloads it. Save the notebook first, run the cell, check the HTML file, and upload it to Canvas.

2. Group exercise: language models through their APIs

Work in groups of three or four. Your group chooses one of the two notebooks. Both start with the same part on APIs: how researchers talk to a language model with a program instead of the chat window.

Everyone runs the notebook on their own laptop, but you work through it together.
Stop at the 💬 Discuss boxes and talk before you go on.
Each member keeps their own notebook, with the names of the group members at the top. You don't hand it in.

Option A, Prompting 1: Are all LLMs the same? Whose history do language models tell?

Five models were asked for "20 important dates in world history" in English, Dutch and Chinese, ten times each. You compare their lists, check an LLM's coding of the events by hand, and ask whose history is missing.

Open in ColabWeek4_Prompting.ipynb

Part	What you do
1. Talking to a model through its API	build a request, read the answer, count tokens and costs
2. The data and how we measure it	150 answers from five models in three languages
3. Are all LLMs the same?	compare the lists with the Jaccard index and tf-idf
4. What is missing?	check an LLM's coding by hand, measure the agreement, count regions
5. Your own prompt	change one thing in the prompt, and save the answers
Reflection	three short answers

Option B, Prompting 2: Frozen in time. Can LLMs forecast world events?

A news video claims that AI models predicted the 2026 Iran war "spookily" well. You test this with forecasts from six models: where each model's knowledge stops, how good its probabilities are, and what happens when a model already knows the answer.

Open in ColabWeek4_Prompting_2.ipynb

Part	What you do
1. Talking to a model through its API	as in option A
2. Where does a model's knowledge stop?	probe the models with dated questions
3. The forecasts	the models' probabilities next to what happened
4. Scoring the forecasts	Brier scores against two baselines; contamination
5. How do the models explain themselves?	a dictionary of continuity words and tf-idf
6. One answer is one sample	the spread over ten runs
Reflection	three short answers

For your group project: the toolkit and a template

Both notebooks use llm_tools.py, a small toolkit for research with language models: ask a model, collect many answers with their details in a CSV file, let a model code texts, check the coding by hand, and compare. The template runs this workflow on your own prompts. Without a key it runs on the data of option A, so you can try it first.

Open the template in ColabLLM_Project_Template.ipynb

An API key (optional, one per group is enough)

Both notebooks work without a key: they use answers we collected beforehand. With a free key you send the requests yourself.

Go to aistudio.google.com → Get API key → Create API key.
In Colab, click the key symbol on the left → Add new secret.
Name it GEMINI_API_KEY, paste the key as its value, and switch on Notebook access.

Never paste an API key into your code. Anyone who sees the notebook can then use it, at your cost, and notebooks are shared all the time. Read the terms of the free tier: Google may use what you send to improve its products, so never send personal data.

The code-along for this week's lecture: text.