"Agents" and "Servers" are generally synonymous, but there is a subtle difference.
Servers represent actual (physical or virtual) servers that BuildMaster can orchestrate. Actions can operate against Servers, files can be transferred from one Server to another, etc.
A "Server Group" is a set of servers that can be treated as a single server.
Agents are the mechanism that BuildMaster uses to communicate with servers, and are tied to a specific protocol (SOAP/Binary/SSH) and/or operating system (Windows, Linux, etc).
Environments are on a "higher level" than Servers and Server Groups; they describe a stage of testing that your application is currently in, e.g. "Integration", "QA Testing", or "Production". Server groups are a way to simplify deployment to clusters, if necessary. Environments can consist of any number of servers and server groups.
To make matters more confusing... you can still work with other servers without an agent, but you're limited to whatever you can do in a normal Windows environment. For example, you can transfer files over UNC paths so long as permissions are set-up, etc. installing an agent and hooking it up to BuildMaster (via External Servers) allows a higher degree of control.