Attempting to Extract Logits Through Fine-Tuning APIs

An investigation into whether fine-tuning APIs leak token probabilities (and why it doesn’t work)

In this blog-post, I will detail an attempted side-channel-esque attack to extract the logprobs for an arbitrary sequence of tokens from an OpenAI model.

Background

At each generation step, a language model assigns a probability to every possible next token given the prompt and the tokens generated so far. A logprob is the natural logarithm of that probability: a value near 0 means the token is highly likely, while a more negative value means it is less likely. Adding the logprobs of successive tokens gives the log-probability of the sequence. The OpenAI generation API exposes these scores through its logprobs parameter.

These scores let us compare candidate responses and measure how a prompt changes the model’s behavior. For example, a jailbreak search can use the probability of a target response as feedback, favoring prompts that make that response more likely. Logprobs also support membership inference attacks (MIA), which try to determine whether a particular piece of text appeared in a model’s training data. The intuition is that a model may assign higher probabilities to text it has trained on. For example, Min-K% Prob scores a passage using the average logprob of its least likely tokens, looking for evidence that even its difficult tokens are familiar to the model. This makes access to token probabilities relevant to privacy auditing as well as jailbreaks and model extraction.

Historically, OpenAI exposed logprobs for only the top-k most probable tokens, but researchers Carlini et al. famously introduced a trick to get arbitrary token probabilities. They did this by exploiting the logit_bias. This parameter allows one to add a value to a token’s logit before the logprobs are computed. Adding a positive bias could bring a target token into the returned top-k, and doing this carefull allows one to read off the logprobs of any token this way. Carlini et al., used this kind of access to recover the output projection layer of production language models, up to symmetries. However, after disclosure, OpenAI patched this “vulnerability” and stopped allowing logit bias to affect the returned top logprobs.

However, I believe that there is a separate side-channel through which one can still recover logprobs given enough resources. This is the fine-tuning API.

The Idea

The fine-tuning API lets you provide a train dataset, training hyper-parameters (learning rate, batch size) and fine-tunes the specified model (probably using LoRA style adapters). The API can be configured to report the per-step training loss, which is

\(\mathtt{training\_loss} \approx -\log\bigl(P(\text{completion} \mid \text{prompt})\bigr)\) Note that if completion is one token long, this corresponds exactly to the logprob of the completion token given the prompt under the current model! This is the exact quantity we want. In order to ensure that the “current model” is the same as the original model, one could set the learning rate to 0. If true, this would be a side-channel information leak. Then, if one wanted to get the logprobs for all vocab tokens under a prompt, one could launch vocab_size number of jobs, with each of them having a dataset of the form

{
  "prompt": "tok_i",
}

One could then set the learning rate to zero, batch size to 1 and read off the train loss. One could also get the sequence level logprobs of an arbitrary sequence like {"prompt","t1,t2,...,tn"} by generating a dataset like

{
  "prompt": "t1",
    "prompt": "t1,t2",
    "prompt": "t1,t2,t3",
    ...
    "prompt": "t1,t2,...,tn"
}

and reading off the per example loss.

However, as I discovered, there are several implementation challenges to be taken care of before this could work.

First Obstacle: LR

The learning rate cannot be set to 0 under the API. However, I set it to the lowest possible value on the OpenAI API (1e-5), and that makes the loss stable across several training steps. I also check this empirically by checking the loss trajectories across shuffles of a dataset and finding them to be identical for small enough datasets.

Second Obstacle: Multiple datapoints

The minimum size of the dataset allowed by the API is 10. A naive way around this is to create a train dataset with 10 repeated examples. However, one can be cleverer and pack multiple sequences into a single dataset and setting train batch size to 1. However, to recover the correct example-wise loss we face another obstacle.

Third Obstacle: Data Shuffling

The fine-tuning API shuffles training data before processing. To map loss values back to our examples, I first needed to reverse-engineer the shuffle order. I find that once the training seed is fixed, the shuffle order is deterministic. To recover this order, I run fine-tuning jobs with simple probes.

I ran 10 fine-tuning jobs, each with 10 examples:

• 9 examples: “What’s the capital of France?” → “Paris” (expected loss = x)
• 1 example: “What’s the capital of France?” → “Berlin” (expected loss = y)

By varying which position contained the “Berlin” example and observing which training step showed the anomalous loss, I recovered the shuffle mapping:

{
  "0": 5,  // Example at position 0 appears at step 5
  "1": 9,
  "2": 0,
  "3": 6,
  ...
}

We can actually do this in O(log n) fine-tuning runs (left as an exercise), where n is the size of the training dataset. Since the shuffle is deterministic for a given seed, we only need to run this probe once.

Does it work?

Now that this is out of the way, we have a way to get the token-wise probabilities for a response. For a query $q$ and response of tokens $t_1,t_2,t_3,\ldots,t_n$ I construct a train dataset of progressively longer samples [($q, t_1$), ($q,t_1:2$),…,($q, t_{1:n}$)]. I then look at the losses for each sample $(l_1,l_2,\ldots,l_n)$ and $-\log p(t_i \mid q,t_{1:i-1}) = i l_i - (i-1)l_{i-1}$, since the losses are averaged across number of tokens. Note that for such a sequence, $il_i$ should be strictly increasing in $i$.

However, I want to understand if $l_i$ actually corresponds to the probabilities of the model. I use the logprobs API as the ground truth to test this.

I hence tried to extract actual token probabilities from the fine-tuning API. For the query “What’s the capital of India?”, I:

1. Got top-10 logprobs from the completions API
2. Created training data with each token as a completion
3. Ran a fine-tuning job and collected per-step losses
4. Applied the inverse shuffle

The values don’t match :( We see that the ranking is roughly preserved, but the magnitudes are completely different. “Delhi” should have had loss of around 0.58, but the fine-tuning API returns 0.19. Similarly “New” should have had a loss of around 0.83, not 10.76.

It seems that something else is being included in the loss calculation.

Hypothesis: Hidden EOS Token

I suspected the fine-tuning API appends an EOS (end-of-sequence) token to each completion. If so, the reported loss would be:

The EOS token’s probability depends on where it appears. After “Delhi”, the model might not expect to stop (it might want to continue with “ is the capital…”), so EOS would have high loss there, driving the reported loss up.

The Progressive Experiment

To test this, I designed a controlled experiment:

1. Generate the model’s own greedy response to “What is the capital of France?”
2. The completions API confirms each token has logprob ≈ 0 (high probability)
3. Create progressive training examples:
   • “The”
   • “The capital”
   • “The capital of”
   • …
   • “The capital of France is Paris.”
4. If there’s no hidden EOS, all losses should be ≈ 0 (since every token is what the model would naturally produce)

Results

The loss is close to 0 when the completion matches exactly where the model would naturally stop. Partial prefixes have high loss despite every token being high-probability according to the logprobs API.

The fine-tuning loss includes the probability of an EOS token at the end of each completion. When we provide a partial sequence like “The”, the model is being told: “The correct response is ‘The’ followed by EOS.” But the model strongly expects to continue after “The”, so EOS contributes high loss.

This means:

The EOS term corrupts attempts to extract clean token probabilities.

A Silver Lining: Sequence level probabilities?

The fine-tuning API reports average per-token loss. Under the EOS hypothesis, for a sequence of N tokens plus EOS:

As N grows, the EOS term contributes $\mathtt{EOS_loss}/(N+1)$. If that term stays bounded, its contribution shrinks, and the reported loss could approximate the average negative log-probability of the response tokens.

I tested this by generating responses with different token limits to the prompt “What are the pros and cons of using linux over windows? Give me an extremely detailed answer.” I create fine-tuning data with different length responses generated, and note the average loss for each response length. I compared the average negative log-probability returned by the completions API with the corresponding fine-tuning loss:

All losses and gaps are in nats per token.

The gap shrinks substantially in this run, reaching a relative error of about 1.23% for the longest response, which seems consistent with the EOS dilution hypothesis.

So there is a limited silver lining: fine-tuning loss can approximate the average negative log-probability of some long responses. This can be used to some extent for jailbreaking or membership inference attacks, however, this seems to be pretty limited in application from the original goal we had.