Robocopy allows you to transfer files with multiple threads. But does more threads mean faster transfers?
In my previous test I have shown that copying/transferring files with /MT is in general faster than not using the parameter.
This leaves us to question: How many threads do we need? And means more threads = more power? Let’s find out.
The setup
For this test I am using a notebook with an Intel Core i7-8550U cpu. This gives me 4 cores with 8 logical processors. The files are being transferred to a NAS over an ethernet connection.
Important remark: This is no lab setup. It’s my daily computer and an active NAS. So the transfer rates might be higher or lower in your environment and some runs might be impacted by processes or users using the system.
Also please note that this is based on an entirely new setup. So you cannot compare the results from the earlier test with this test (And you will see that there are a lot of differences).
Similar to the old test, I have prepared 3 scenarios with a volume of ~20GB each:
- 1 single file with 21.474.836.480 bytes
- 2.701 image files with 21.504.029.144 bytes (7-8 MB / each)
- 43.866 text files with 21.557.901.592 bytes (400-600KB / each)
I used the following command to copy the files to the NAS:
robocopy.exe sourcepath destinationpath *.* /W:1 /R:1 /NFL /MT:n
/W and /R means that robocopy retries reading/writing files only once. I checked after each run if all files have been transferred without issue.
/NFL means that the files transferred will not be listed. The keeps my command prompt clean.
/MT defines the number of threads. For runs with “No MT” I completely omitted this parameter.
I started each scenario with the following thread count: No MT, 1, 2, 4, 8, 16, 32, 64, 128. And each of these combination ten times. This means a total of 270 copy jobs.
For each run I logged the duration using the column “Copied” and “Extras” plus the sum of both.
The results
Single file
The results of the earlier test were pretty clear: Multi threading saved you up to 50% of the run time.
Copying a single file with more or less threads gave me a pretty inconsistent results. Using 4 threads made the entire process even worse. But all in all the duration stayed more or less the same.
Image files
The initial test without /MT takes ~280 seconds. Using 1 thread only worsened the result. You need 54 seconds (almost a minute!) longer to copy the files. After that the run times decrease significantly: 8 threads save you over a minute (280 vs 215 seconds). And after 32 threads the results decrease only by a few seconds.
Text files
This test had the biggest impact on the duration with more threads.
Staring without /MT tool 1314 seconds and got -again- worse with 1 thread. But after that the durations dropped very fast: 4 threads saved you more than 14 minutes and 128 threads take another 100 seconds less!
Summary / Conclusion
Using /MT is in general a good idea. You might not benefit from it, but it also won’t hurt you (unless you are using 1 thread only. Bad idea!). Especially small files in large numbers copy faster with more threads. For my setup, more threads worked almost always better, even if I used more threads than my CPU could handle at the same time.
But keep in mind, that this might affect you background processes or services if you are running robocopy with a lot of threads on a production server!
Thank you for this! I need to move 6TB of files to a NAS and really was curious. I appreciate your scientific approach
I’m not surprised that specifying 1 thread resulted in worse performance as all the technical documentation I’ve read said that the default, i.e. without specifying the MT parameter, value is 8 threads. Therefore you’d think that anything under 8 threads would be worse. The question in my mind is how far can you push it. I’ve read somewhere that the number of maximum threads is related to the number of cores but I can’t remember if that’s physical or virtual.
Slight correction, robocopy by default does not run multithreaded without the /mt flag.
That documentation referencing the 8 thread ‘default’ applies to running the /mt flag without specifying a number of threads.
‘robocopy /mt’ is equivalent to running ‘robocopy /mt:8’
But ‘robocopy’ with no /mt flag will not perform any multithreading.
Thanks for information on the MT switch. Very helpful
A more subjective test would be to see how many threads it takes to get a network admin to knock on your door and tell you to slow it down 🙂
What is the max capacity on your network card? 1GB? 10GB? Once you saturate your network throughput, more threads aren’t really going to help much. This is nice, but for this information to be more worthwhile, would be nice to see your network throughput & CPU utilization during each of the tests.
Good idea. I’ll check if I can set this up in the future.
This is going to be highly dependent on the storage platform you are running it on. While I could go WAY more into detail here is what should be shared.
Spinning disk (5K, 7K, 10K, 15K) –> As the number of threads goes up and size of your files goes down so will the number of disk head repositions on the source and level of fragmentation on the destination. (Atleast until the size of the file is under the size of the allocation block of the volume). Technical capabilities like native command queuing start to play a role.
SSD –> SSD excels with lots of threads and fragmentation does not matter like spinning disk because there is no penalty from fragmentation for most workloads. (Some can even benefit). The loss here is often the size of the data (vs size of the file) that needs to be written.
Tiered storage, object storage, NAS, … –> Depends on the platform.
Source – I do this for a living but if you want more info suggest books by EMC.