When you say delete, do you mean remove from a workflow such that the build never travels into that environment? If so, just delete the workflow step from the workflow.
If you actually want to keep the workflow step but remove all the action groups, just delete them one-by-one.
by
Karl