Problems about Usage of SyncSN

[stillwaterman]

Very nice work! I try to use your train code in face_recognition, but I met some problems. Frist, rank = int(os.environ['RANK']) and world_size = int(os.environ['WORLD_SIZE']) don't have values, so I added some code os.environ['RANK']=str(0), os.environ['WORLD_SIZE']=str(4). Is that right? Second, my code is stuck at dist.broadcast, it doesn't have any error message, just stuck. Could you give me some advice

[JiaminRen]

What problem? I will give a example soon.

[stillwaterman]

@JiaminRen my code is stuck at dist.broadcast, no error message, backend is nccl. Do you test the train code or what configuration I didn’t do

[JiaminRen]

which task did you test? imagenet or face recognition?

[stillwaterman]

@JiaminRen I just tried to imitate your train code in face recognition to use SyncSN in my code, but
I didn't succeed. I met two problem, frist, rank = int(os.environ['RANK']) and world_size = int(os.environ['WORLD_SIZE']) don't have values, so I added some code os.environ['RANK']=str(0), os.environ['WORLD_SIZE']=str(4), second is dist.broadcast

[JiaminRen]

Have you changed any code? Just running the script face_recognition/train.sh will be ok.

[stillwaterman]

@JiaminRen I quickly test the face_recognition train.py, unfortunately I met the same problems. I think maybe some system configurations I missed.

[stillwaterman]

@JiaminRen my system is ubuntu18.04 and I use ananconda to install pytorch, program is stuck at dist.broadcast

[JiaminRen]

This is a distributed framework, and it should be run on multi-gpus by using torch.distributed.launch.

[stillwaterman]

Thanks, torch.distributed.launch can solve problems. But sync way consumes a lot of GPU memory, always out of memory

[stillwaterman]

Sorry to bother you again. Actually, when I was using SyncSN, I got some different errors. I tried to imitate the way you used in train.py, but my model outputs NaNs, which will not happened in SN. Another error is subprocess.CalledProcessError: Command returned non-zero exit 1. Do you have any idea? Thanks