# `Tinkex.TrainingClient.DataProcessor`
[🔗](https://github.com/North-Shore-AI/tinkex/blob/v0.4.0/lib/tinkex/training_client/data_processor.ex#L1)

Data chunking, numbering, and tensor operations for TrainingClient.

This module handles:
- Chunking training data based on size limits
- Estimating chunk sizes using byte heuristics
- Building placeholder gradients for custom loss
- Extracting target tokens from loss function inputs

# `allocate_request_ids`

```elixir
@spec allocate_request_ids(non_neg_integer(), pos_integer()) ::
  {[pos_integer()], pos_integer()}
```

Allocate sequential request IDs for a batch of requests.

Returns `{[id1, id2, ...], new_counter}` where the IDs are consecutive
starting from the current counter.

# `build_placeholder_gradients`

```elixir
@spec build_placeholder_gradients([Tinkex.Types.Datum.t()]) ::
  {:ok, [Nx.Tensor.t()]} | {:error, Tinkex.Error.t()}
```

Build placeholder gradients (zeros) for custom loss computation.

Creates zero-filled tensors matching the shape of target_tokens for each datum.
These are used as placeholder gradients before the actual loss computation.

# `chunk_data`

```elixir
@spec chunk_data(list()) :: [list()]
```

Chunk data into manageable pieces based on size and byte limits.

Ensures no chunk exceeds:
- 1024 items
- 5000000 total estimated bytes

# `fetch_target_tokens_tensor`

```elixir
@spec fetch_target_tokens_tensor(Tinkex.Types.Datum.t()) ::
  {:ok, Nx.Tensor.t()} | {:error, Tinkex.Error.t()}
```

Extract target_tokens tensor from a datum's loss_fn_inputs.

Supports both TensorData and Nx.Tensor formats.

---

*Consult [api-reference.md](api-reference.md) for complete listing*