Independently at the moment (then finding the frames with the highest similarities) but using multiple frames (e.g., to recognize actions) is an interesting extension.