Deploy artifact containing many small files

Currently when deploying artifact, it seems to extract the artifact first on server and send the files individually over to agent.

I have a case where the latency between the bm server and agent is quite slow, copying 1 bigger file is much faster than copying many small files, let alone the artifact is compressed.

The deploy task occasionally results in
Unhandled exception: System.IO.IOException: The process cannot access the file 'xxx' because it is being used by another process. Probably caused by this, as there shouldn't be other processes locking newly transferred files.

Would it be possible to transfer the artifact to the agent first, and let the agent handle the artifact from there? Should be faster and hopefully not causing the occassional error.

Product: BuildMaster
Version: 5.3.8

The Deploy Artifact Operation/Action does a differencing of the files in the target (remote) folder, and the files contained in a zip, and then sends those over in stream. So, it's quite fast compared to sending a whole zip, or the Transfer Files Operation/Action. But you could do those too.

You should definitely investigate that file in use. Unless you have simultaneous deployments running against the same folder in BuilldMaster, another program is locking the files. It might be an anti-virus/malware tool that's scanning a changed file. Also, if you haven't already, take a look at Avoiding Process Cannot Access File Errors with ASP.NET Deployments

I'm not sure how the differencing works. My doubt is usually the deploy artifact would clear the target directory first, so does that mean the differencing would be effectively copying full files? I observed the files appearing in the folder one by one. For some larger files (a few mb), the size increases gradually. If the files contained in a zip being transferred as a whole, shouldn't they be appearing almost all at once?

There might be ant-virus/malware tools scanning changed files, especially .exe files. Unfortunately, sometimes the .exe files is relatively large that their size increases gradually, therefore probably locked by the anti-virus tools while the deployment is still going.

I wasn't deploying asp.net files, only plain console. So stopping AppPool is not needed.

If the files contained in a zip being transferred as a whole, shouldn't they be appearing almost all at once?

It's like downloading a file from your browser... the files are sent are written to disk as they are being streamed.

Unfortunately, sometimes the .exe files is relatively large that their size increases gradually, therefore probably locked by the anti-virus tools while the deployment is still going.

That could happen; it's possible the AV tool is acquiring a lock between chunks. In this case, your best bet would be to disable the AV tool or, if not possible, then use the zip method you described.