The clients' public keys are uploaded to the switch via TFTP or SFTP. Just remove the unwanted public key(s) from the key file and upload it again, overwriting the switch's local file. If you don't want any public keys, upload an empty file.
Either
copy tftp pub-key-file <ip-address> <filename>
or
SFTP to /ssh/mgr_keys/authorized_keys