"Agents" and "servers" are generally synonymous - an agent can be a lightweight service that BuildMaster sends requests to in order to run actions on a remote server (or as of v3.1, an extensible component on the BuildMaster server itself that interacts with remote servers through other channels). Environments are on a "higher level" than servers; they describe a stage of testing that your application is currently in, e.g. "Integration", "QA Testing", or "Production". Server groups are a way to simplify deployment to clusters, if necessary. Environments can consist of any number of servers and server groups.
I note that you can still work with other servers without an agent, but you're limited to whatever you can do in a normal Windows environment. For example, you can transfer files over UNC paths so long as permissions are set-up, etc. installing an agent and hooking it up to BuildMaster (via External Servers) allows a higher degree of control.
Agent Requirements: Windows Server 2003 or later with at least .NET 2.0 installed for the IIS version, .NET 3.5 installed for the self-hosted agent server, or Linux with SSH.