The The Face Manager (https://www.photools.com/help/imatch/people-face-manager.htm) should do this very quickly.
Switch to unconfirmed, <Ctrl>+<A>, click Ignore.
Maybe, instead of ignoring, if you are really not interested in a face, delete the face annotation instead.
This reduces the number of faces IMatch has to manage.
Whatever works best for you.