Performance issues with shutil.copytree on windows

[Marc-Pierre-Barbier]

Feature or enhancement

Proposal:

I noticed that Conan seemed to softlock while building clang-tidy on Windows, and I wondered why. After digging around for a while I found that _copy_sources call shutil.copytree which seems to get stuck for a long time. It's technically not stuck, but simply slow.

To compare I wanted to try robocopy so that I can have performance numbers, so I patched installer.py._copy_sources :

                start_time = time.time()
                shutil.copytree(source_folder, build_folder, symlinks=True)
                end_time = time.time()
                print(end_time - start_time)

This runs for 1329.195054769516s

                start_time = time.time()
                out = subprocess.run(['robocopy', '/ndl', '/nfl', '/sl', '/S', source_folder+'\\', build_folder+'\\'])
                assert out.returncode <= 1 # 1 and 0 are not errors anything else should contain an error
                end_time = time.time()
                print(end_time - start_time)

this runs for 92.37016916275024s

The copied folder contains just the llvm source code as shipped in their release page. This is not a theoretical workload.

So robocopy is significantly faster, finishing in 1.5m, while shutil needed 22 minutes.
This solution is obviously not clean and takes a lot more space on disk (no longer creating symlinks) but it does show that there is indeed a big performance issue with shutil with high file count copies.

I will update this issue tomorrow with performance numbers for symlinks=false.

Has this already been discussed elsewhere?

It was already discussed on Discourse a while ago

Links to previous discussion of this feature:

At the time the only proposition was using multi threading. I did not benchmark their solution as at the time it was found inadequate but maybe now that the GIL is less of an issue it is worth considering again.

#124117
https://discuss.python.org/t/significantly-improve-shutil-copytree/62078/25

[picnixz]

• Can yoh check the difference of shutil between windows and linux instead of comparing it to another copy utility?
• How large is the srcdir? is it an unexpected slow down or is it scaling as expected?

[Marc-Pierre-Barbier]

• Can yoh check the difference of shutil between windows and linux instead of comparing it to another copy utility?
  I initially did that to show that the bottleneck wasn't simply the drive io.

This time I benchmarked with a full LLVM source dir: 116916 Files totalling 1.47GB
Linux is using WSL on the same machine as the windows numbers.

Linux:
shutil.copytree takes 18.3seconds with symlink=True
and 12.18s with it at symlink=False.
Windows (shutil.copytree symlinks = True):
1811.7s
Robocopy:
204.5s

* How large is the srcdir? is it an unexpected slow down or is it scaling as expected?

116916 Files totalling 1.47GB

The performance gap is too big to just be a python/windows problem, here is my hypothesis:

1. Linux is not being slowed down by windows defender
2. Windows defender makes every individual file operation slow
3. robocopy is faster because multi threading allow for multiple files to be scanned by the AV at once.

Sadly, on my work computer, I don't have control over antivirus settings.

During the copy operation (shutil) I can see a constant 4% cpu load caused by the av process. While using robocopy the CPU load increase to 5-8% (varying a lot) it's not a big in crease, but maybe it's enough to explain the difference ?

To prove/disprove my hypothesis, I will try to multithread shutil.copytree to see if it can match robocopy

[Marc-Pierre-Barbier]

with ThreadPoolExecutor() as executor:
  def threaded_copy(src:str, dest: str):
    executor.submit(lambda : shutil.copy2(src,dest))
  
  shutil.copytree(source_folder, build_folder, copy_function=threaded_copy)

Without patching shutil the above code lower the time needed for the copy to 150

Since the threadpoolexecutor default to 5xcore count (in my case 5x22) it's way more cores than robocopy use and significantly mitigate the performance impact of the AV.

This code snippet could be used in the shutil.copytree as it wrap shutil._copytree.

If you like the idea, let me know and I will open a PR.

[picnixz]

I think it is better to include a recipe rather than changing the implementation itself (while it is slow, stability needs to be retained as much as possible). At least, it would be easy to add that recipe to the docs (while changing the implementation adds a maintenance burden that we may not want).

cc @gpshead @serhiy-storchaka