Thanks!
Those links were helpful. My main misunderstanding was regarding the uploaded certificate in the IMC config.
I assumed it was supposed to be a client cert issued to the IMC server, NOT the cert created using the LDAP over SSL template and issued to the DC
Now it works
EDIT: downside, I'm running active and standby. The standby gets its configuration for this through the nightly backup as there is no option to configure much on the standby server. The regular LDAP config gets synced, but the LDAP over SSL does not get correctly configured. So make sure you have a local admin account configured as no authentication sever will be available.