> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Input Data

> Learn how to configure your input data for preprocessing—whether you're working with a single directory of data or organizing large datasets into subsets.

You can configure **local** or **Hugging Face** data as input for preprocessing. In this guide you'll learn how to define your data source, specify optional parameters like subsets or splits, and structure your config file to support flexible, scalable preprocessing workflows.

## Local Data

* Set `type` to `local`.

* Use `source` to provide the path to the input directory.

For example:

```yaml theme={null}
  setup:
    data:
        source: "/input/dir/here"
        type: "local"

    mode: "pretraining"
    output_dir: "./output/dir/here/"
    processes: 1

```

### Preprocess Subdirectories

You can optionally preprocess subdirectories within your input directory as **separate datasets**. This enables more flexible data management for large-scale pretraining tasks.

There are two supported options:

* Use `top_level_as_subsets: True` to automatically treat **each top-level folder** in your input directory as a separate dataset. Each top-level directory is treated as a subset and a separate output folder will be created under `output_dir` with its respective preprocessed HDF5 files. Defaults to `False` if not specified.
* Use `subsets: [list]` to manually specify which subfolders to preprocess. Only the folders listed in subsets will be preprocessed and each subset will have its own output folder under `output_dir`.

Use the tabs to view examples:

<CodeGroup>
  ```yaml Top-Level Subsets {5} theme={null}
  setup:
      data:
          source: "input_dir"
          type: "local"
          top_level_as_subsets: True
      mode: "pretraining"
      output_dir: "./output_dir"
      processes: 1
  ```

  ```yaml Specific Subfolders {5} theme={null}
  setup:
      data:
          source: "input_dir"
          type: "local"
          subsets: ["subset_1", "subset_2" ...]
      mode: "pretraining"
      output_dir: "./output_dir"
      processes: 1
  ```
</CodeGroup>

## Hugging Face Data

* Set `type` to `huggingface`.

* Use `source` to specify the dataset name from the Hugging Face hub.

* Use `split` to specify the dataset split.

The preprocessing pipeline passes these parameters to the Hugging Face `load_dataset` API.

<Note>
  When calling the API, parameters are passed as keyword arguments and they must conform to the specifications outlined by HuggingFace. Refer to the `load_dataset` documentation [here](https://huggingface.co/docs/datasets/v2.20.0/en/package_reference/loading_methods#datasets.load_dataset).
</Note>

For example:

```yaml Hugging Face Data {3-5} theme={null}
setup:
  data:
      source: "stanfordnlp/imdb"
      type: "huggingface"
      split: "test"
      cache_dir: "path/to/cache_dir"
      ...other parameters accepted by HuggingFace ``load_dataset`` API...


  mode: "pretraining"
  output_dir: "./output/dir/here/"
  processes: 1

```

<Tip>View example configs for various use cases [here](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/data_preparation/data_preprocessing/configs). </Tip>