Troubleshooting

• Cannot load Cerebras checkpoints in GPUs

  • Work around

• Custom PT training script spawns multiple compile jobs

  • Observed Error

  • Explanation

  • Work around

• Loss compilation issues with Autogen

  • Custom loss functions with AutoGen

  • Improving loss function performance

• Error parsing metadata

  • Observed Error

  • Explanation

  • Work around

• Error Receiving Activation

  • cerebras.appliance.errors.ApplianceUnknownError: Ran into error while receiving activation tensor <custom-call …>

• Failed mount directory during execution

  • Observed Error

  • Work around

• Failing to automatically load checkpoints

  • Explanation

  • Work around

• Failure to trace due to functionalization error

  • Observed Error

  • Explanation

  • Work around

• Input Starvation

• Out of memory errors and system resources

  • Determining if your job is queued

  • Determining if job failed because of an OOM error

  • Determining if job failed because of system could not fit requested memory

  • Troubleshooting OOM errors

• Model is too large to fit in memory

  • Observed Error

  • Causes and Possible Solutions

• ModuleNotFoundError

  • ModuleNotFoundError: No module named <’_bz2’, ‘_sqlite3’>

  • ModuleNotFoundError: No module named <…>

• Numerical issues

  • Observed Error

  • Explanation

  • Work around

• Throughput spike after saving checkpoints

• Training fails when logged-in as root

  • Observed Error

  • Explanation

• Vocabulary Size Troubleshooting

  • Large vocabulary size

  • Small vocabulary size