Job Priority
Learn how jobs are prioritized on a Cerebras cluster.
Job scheduling is determined by assigned priorities rather than submission time. Jobs are categorized into four priority buckets, ranging from P0 (highest) to P3 (lowest). Within each bucket, jobs are ranked by numerical priority values, where lower numbers indicate higher priority. Scheduling follows these rankings within designated queues.
Submit Jobs with Priority
Users can specify a job priority at submission using the --job_priority
flag. For example:
Valid inputs are p1, p2, and p3, corresponding to default values of 199, 299, and 399, respectively. If unspecified, jobs default to P2 (299). Learn more about the additional arguments for launching jobs here.
Update Job Priority
Users can view and adjust a job’s priority post-submission using the csctl tool.
-
Only admin users and users with the same GID can modify a job’s priority.
-
Only admin users can access the P0 bucket (0 to 99).
-
Completed jobs cannot have their priorities updated.
Monitoring Job Priorities
Use the csctl tool to view job priorities:
When listing jobs, you will see both priority values and their corresponding buckets. Jobs are first sorted by priority and then by age. Higher-priority jobs appear first, and within the same priority level, newer jobs are listed before older ones.
Priority Continuity
Jobs stemming from the same invocation inherit the workflow’s priority. Updating a workflow’s priority affects all associated pending and subsequent jobs, ensuring consistent priority across a workflow.
Additional usage examples
Update a job’s priority to a different bucket:
This updates the job’s priority to P2. Confirmation is displayed:
List jobs with their priorities:
Displays a list of jobs along with details such as type, priority, age, duration, phase, system, user, labels, and dashboard status. For example, it shows that: