Just noting that the Turing series of cards and later (which isn't just RTX, but also the GTX16* the the T4) does have specialised hardware which helps with the video tracking side of this.

They have hardware optical flow[1] which doesn't seem widely known, but would absolutely help with parts of this - especially the reframing.

[1] https://developer.nvidia.com/opticalflow-sdk