I’ve got following problem:
I am training a TensorFlow 2.6 Matterport MaskRCNN-Port on my RTX3090. I’ve installed CUDA 11.4 and CUDNN.
The GPU is shown in Tensorboard.
When I’m training the GPU is idle on 0% GPU-util and suddenly spikes after a time, processing a batch of images. The CPU (64 core) is going 100% on ~2 cores beforehand.
Why is there as much idle time? Tensorboard says there is no input-pipeline "lag".
Linux, RTX3090, 64-core CPU (EPYC), CUDNN 8.0.2, CUDA 11.4.1