[ER] Stuck at "Please wait while the system is starting up"



When attempting to access the web console the start-up screen pictured below appears. This screen does not disappear despite waiting several minutes.





The most common reason for this behaviour is a corrupt database on the Master Server. Database corruption is rare, but can occur when:

  1. The Master Server was shut down while it was still busy and utilizing important data.
  2. The Master Server has run out of disk space resulting in file corruption.



Before completing any of the below steps, it is important to determine which of the above reasons caused the original failure. Please check the available disk space; if it is low then please follow the instructions to increase your disk space. As a rule of thumb we would generally advise doubling the current amount of space, so if your Master Server has 20GB of space then it should be resized to 40GB (and so on).

It is important to note that recovery of the Master Server may not be possible if the corruption is severe - in which case the only other alternative is to rebuild your entire Master Server.

Follow the instructions below to attempt the recovery procedure:

  1. Login to your ER Master Server Terminal console as 'root'
  2. Stop the ER service using the command; /etc/init.d/er2-master stop
    This step may take a while, we recommend waiting for 5 minutes before going to the next step
  3. Run the command; er2-recovery -i /var/lib/er2/db/root.kct -r -w /var/lib/er2/db/recovered.kct

  4. If it looks like all has gone well, run the following commands in sequence;
    mv /var/lib/er2/db/root.kct /var/lib/er2/db/old.kct
    mv /var/lib/er2/db/root.kct.wal /var/lib/er2/db/old.kct.wal
    mv /var/lib/er2/db/recovered.kct /var/lib/er2/db/root.kct
    chown erecon:erecon /var/lib/er2/db/root.kct
    chmod go-r /var/lib/er2/db/root.kct
  5. Start the ER service again using the command; /etc/init.d/er2-master start


  6. Open up the ER web console again and allow for 5-15 minutes for your Master Server to start up (a refresh may be needed)
  7. Once recovery has been fully confirmed, the old database files can be removed:
    rm /var/lib/er2/db/old.kct
    rm /var/lib/er2/db/old.kct.wal

Important notes

The er2-recovery process attempts to read as much as possible from the original database, however it is possible that some records may be lost or missing. 

To prevent such problems from happening again, it is crucial to make sure that the Master Server is gracefully shutdown using the following commands:

/etc/init.d/er2-master stop
shutdown -h now

We also highly recommend backing up your Master Server by taking a VM snapshot or using the integrated ER backup tool which you may learn more about in this article.


All information in this article is accurate and true as of the last edited date.


0 out of 0 found this helpful



Please sign in to leave a comment.