We are very pleased to announce that the outage, we were struggling with since Tuesday (March 25) is over.
On March 25 TimeCamp notified an intermittent outage. It was the result of technical issues with our servers’ capacity. They slowed down and were unstable what was caused by corrupted database. We needed to disable all the records in the tables. Because there was a problem with the replication server the only option to resolve this was to move the entire database to other servers. Due to a big amount of SQL data this process took about two days.
During this time all our users had no access to their TimeCamp accounts, but all the data from desktop application were registered on their computers.
The system is up and running again. To make this happen quicker we decided not to import all historical data from desktop application right on as it is the biggest SQL table in TimeCamp. It may take a couple of days till all the data are returned. The records registered since the beginning of the year till now should be returned before Monday. The rest will be delivered in the next days.
The Lessons Are Learned
This situation showed us that:
– We should always have a script that monitors if all backup systems are up and running (we have daily full backups and replication server)
– We should never rush anything – we have made some mistakes that cost us hours of interrupted TimeCamp,
– We shouldn’t be short-sighted – we always have to calculate how long can things take. If we do so, we can see other, quicker possibilities.
We will do our best to not let such situations happen again.
We are very sorry for all the inconvenience.
Thank you for your patience and being part of TimeCamp community