The Custom Worker Container Workflow provides seamless support for importing user-specific dependency packages in the Python environments into Cerebras appliance. With this feature, you do not need any special handling. In the event where the custom worker container failed to be created, a fallback policy will take into effect by mounting the site packages from the user node Python virtual environment to the worker environment via a predefined NFS-based cluster volume. Both the custom worker container feature and the venv mounting fallback policy have been enabled by default. If you would like to disable the features, there are two Cerebras-specific options that can support that. We call these options debug_args.
  • To disable the custom worker container feature, the option is named debug_args.debug_usr.skip_image_build. Setting this option to True will disable this feature.
  • To disable the venv mounting fallback policy, the option is named
debug_args.debug_usr.skip_user_venv_mount. Setting this option to True will disable the fallback policy.

​
Procedure

To disable custom worker container workflow, follow the steps below:
  1. Save the following script as debug_args_writer.py on any accessible directory on the user node:
from cerebras.appliance.run_utils import write_debug_args
from cerebras.appliance.pb.workflow.appliance.common.common_config_pb2 import DebugArgs

debug_args = DebugArgs()
debug_args.debug_usr.skip_image_build = True
debug_args.debug_usr.skip_user_venv_mount = True

out_file = "debug_args.out"
write_debug_args(debug_args, out_file)
  1. Run it inside a Cerebras virtual environment you have setup on the user node, see Setup and installation:
(venv_cerebras_pt) $ python debug_args_writer.py
This script will produce a file debug_args.out. You can then add the additional option when invoking run.py as follows: 
(venv_cerebras_pt) $ cszoo fit <other args> --debug_args_path debug_args.out
Currently, the custom worker container workflow only accounts for Python dependencies that were pip installed on the user node and assumes Internet and PyPI access from the workers in Cerebras appliance. If workers do not have access to the Internet or PyPI, the workflow will fail and the fallback policy will take into effect.