Security

Hello All,

Due to a series of unfortunate events, both our publisher and subscriber Clearpass servers are stuck in emergency mode. We tried using our appadmin password at the CLI to do some recovery stuff but it wasn't working. I tried to call Aruba TAC and they tried using the eTIPS password I have seen all over the internet, but told me that's all they can do and I have to rebuild completely. 

I'm fairly new to Clearpass so I just have to ask if anyone else can confirm or deny if the only breakglass option for this account is an 8 character or less default password and there are no other recovery options. I've not gotten far enough to get backups going or even consider the best way to back up one of these servers so my best guess is I'm completely hosed and there's no way to bring anything back online if they are correct. 

It depends a bit what emergency mode is; not sure what it exactly means. If it's the Linux system not able to find the boot files, you may try a boot recovery system for Linux/RedHat, but that doesn't work if you enabled full-disk encryption; and if you haven't it's not supported but may give you access once to create a backup.

If the system boots through to the command prompt login, TAC should be able to access support mode and reset the passwords. But there may be requirements that the system is network accessible. If you explain the 'series of unfortunate events' in more detail, TAC may find a better way to regain access as it greatly helps to understand how you got into this situation.

BTW, the appadmin password is only 'eTIPS123' for the first login. During the initial configuration, you need to change it to something else, which also becomes the WebUI admin password, and the Cluster Password, If you still have access to the WebUI, you can change the Cluster Password, the appadmin password will be set to the same. So, if you changed the password initially, or changed the cluster password later on, you may try that one to get back in.

Hello All,

Due to a series of unfortunate events, both our publisher and subscriber Clearpass servers are stuck in emergency mode. We tried using our appadmin password at the CLI to do some recovery stuff but it wasn't working. I tried to call Aruba TAC and they tried using the eTIPS password I have seen all over the internet, but told me that's all they can do and I have to rebuild completely. 

I'm fairly new to Clearpass so I just have to ask if anyone else can confirm or deny if the only breakglass option for this account is an 8 character or less default password and there are no other recovery options. I've not gotten far enough to get backups going or even consider the best way to back up one of these servers so my best guess is I'm completely hosed and there's no way to bring anything back online if they are correct. 

Thanks as always for contributing Herman, I can explain in more detail now and I have found a solution, and hopefully can get 1 more confirmation. 

What occurred: yesterday we had a system wide network failure in my office. Every server and client was disconnected. When the connections were restored our virtualized Clearpass servers were unable to recover, so I had them reboot. (Clearpass wasn't the only one to have recovery issues, so we were restarting servers that stumbled). I am attaching a screenshot of the last bit of what happened during the initial boot. After these errors, the last paragraph repeated itself. I'm going to type this in specifically so the internet search can find it in the event someone else runs through this:

You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode. sulogin:tcgetattr failed: input/output error

I did see similar errors about "Emergency Mode" based linux OS responses that required a disk scan to fix the error. The solution to fix the error I found was to edit the boot loader. I've attached an image of the boot loader choice I was trying to edit using the "Press 'e' to edit the selected item..." option. I was able to fix a different VM using this option but Clearpass wants a user and password to edit this field and that is where I was stuck. Our appadmin + password combo did not work here.

Outstanding question: Should appadmin work here?

This is where support had given up. They did expect appadmin to work here. We have tested appadmin since restoring the system and verified we were using the correct username / password.

The Fix: 

Disclaimer: Support is aware I did this, but the technician did not completely understand what I had done. I only took these steps so I could get a better backup. Support is watching my servers with me to test stability (no current issues) but this is not in a technical article yet. Do not proceed without consulting them.

Since my server was already broken beyond support

1. I used a boot loader repair tool, like Herman's suggestion. I don't want to link or recommend a specific tool as this is at your own risk, but I used a supergrub tool personally.
2. I took a snapshot.
3. I needed to boot into the ISO file, but had no disk drive, so I added one to my VM. 
4. I was unable to get this to work while in EFI boot mode. So AFTER TAKING A SNAPSHOT I opted to change the hard drive from EFI boot to BIOS boot on my virtual box. 
5. I changed the boot order so I could run the grub loader tool and boot to the CD. Supergrub scans for various boot loaders, so I ran a scan to find all available.
6. I was presented with 2 options (screenshot attached) that had a similar name to the normal load. I don't know what the rescue one is for, so I highlighted the regular one and typed 'e' to edit.
7. I scrolled to the bottom of the grub loader where they have the Linux boot command (image attached). After the long line loading linux I added:
   fsck.repair=yes
   to run a file check on bootup. 

The file check ran, repaired whatever was seen as corrupted, and Clearpass booted. I then rolled back my bios to boot in EFI and the server was up.

It depends a bit what emergency mode is; not sure what it exactly means. If it's the Linux system not able to find the boot files, you may try a boot recovery system for Linux/RedHat, but that doesn't work if you enabled full-disk encryption; and if you haven't it's not supported but may give you access once to create a backup.

If the system boots through to the command prompt login, TAC should be able to access support mode and reset the passwords. But there may be requirements that the system is network accessible. If you explain the 'series of unfortunate events' in more detail, TAC may find a better way to regain access as it greatly helps to understand how you got into this situation.

BTW, the appadmin password is only 'eTIPS123' for the first login. During the initial configuration, you need to change it to something else, which also becomes the WebUI admin password, and the Cluster Password, If you still have access to the WebUI, you can change the Cluster Password, the appadmin password will be set to the same. So, if you changed the password initially, or changed the cluster password later on, you may try that one to get back in.

Hello All,

Due to a series of unfortunate events, both our publisher and subscriber Clearpass servers are stuck in emergency mode. We tried using our appadmin password at the CLI to do some recovery stuff but it wasn't working. I tried to call Aruba TAC and they tried using the eTIPS password I have seen all over the internet, but told me that's all they can do and I have to rebuild completely. 

I'm fairly new to Clearpass so I just have to ask if anyone else can confirm or deny if the only breakglass option for this account is an 8 character or less default password and there are no other recovery options. I've not gotten far enough to get backups going or even consider the best way to back up one of these servers so my best guess is I'm completely hosed and there's no way to bring anything back online if they are correct.