polyaxon
I have two problems with this approach:
\n- \n
- I need to copy the env files to all our git repos, which means if I make a change I need to perform several pull requests \n
- I need to tell the data-scientits to pull the last commit, sometimes that's not possible because they can not merge/rebase the changes. \n
Based on those two issues, in the end we tell data-scientists to just use:
\nenvironment:\n nodeSelector:\n nodes: large-pool\n...\nrun:\n ...\n container:\n resources:\n limits:\n cpu: 3000m\n memory: 6000Mi\n requests:\n cpu: 2000m\n memory: 4000MiWhich is error prone and confusing for them, and make the files bigger and difficult to change.
\nAny elegant way to abstract this type of configuration from the data-scientists?
","upvoteCount":1,"answerCount":1,"acceptedAnswer":{"@type":"Answer","text":"We have already shared a resource on how to configure the environments in this guide
\nif you are using multiple git repos and you do not want to replicate the yaml files in all repos you can register those files as presets:
\n- \n
- Users will be able to use
--presets machine1or--presets=env1\n
Note that in the example in that link, it shows that it defines a queue but you do not have to define a queue, a preset is just any YAML file that can be used with the override operator -f main.yaml -f override1.yaml -f override2.yaml in this case override1.yaml and override2.yaml it can be saved as organization presets using the UI.
More info from the intro section about presets and the UI section
\nAlso, when you define presets you can use them directly on the operation or component
\npresets: [preset1, preset2]This is similar to the CLI command
\npolyaxon run -f polyaxon.yaml --presets preset1,preset2I would like to configure Polyaxon in a way to avoid asking data-scientists to configure pre-emptible node-pools or request TPUs on their own #1484
-
I am currently defining some machines configuration using polyaxon run -f polyaxonfile.yaml -f machine-env1.yamlI have two problems with this approach:
Based on those two issues, in the end we tell data-scientists to just use: environment:
nodeSelector:
nodes: large-pool
...
run:
...
container:
resources:
limits:
cpu: 3000m
memory: 6000Mi
requests:
cpu: 2000m
memory: 4000MiWhich is error prone and confusing for them, and make the files bigger and difficult to change. Any elegant way to abstract this type of configuration from the data-scientists? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
We have already shared a resource on how to configure the environments in this guide if you are using multiple git repos and you do not want to replicate the yaml files in all repos you can register those files as presets:
Note that in the example in that link, it shows that it defines a queue but you do not have to define a queue, a preset is just any YAML file that can be used with the override operator More info from the intro section about presets and the UI section Also, when you define presets you can use them directly on the operation or component presets: [preset1, preset2]This is similar to the CLI command polyaxon run -f polyaxon.yaml --presets preset1,preset2 |
Beta Was this translation helpful? Give feedback.
We have already shared a resource on how to configure the environments in this guide
if you are using multiple git repos and you do not want to replicate the yaml files in all repos you can register those files as presets:
--presets machine1or--presets=env1Note that in the example in that link, it shows that it defines a queue but you do not have to define a queue, a preset is just any YAML file that can be used with the override operator
-f main.yaml -f override1.yaml -f override2.yamlin this caseoverride1.yamlandoverride2.yamlit can be saved as organization presets using the UI.More info from the intro section about presets and the UI section
Also, w…