Service outage

Incident Report for Lyyti

Postmortem

Today on Friday 2019-10-04 at 11:11 UTC, we experienced a short but complete outage of all our web services. Service was restored 15 minutes later at 11:26 UTC.

The cause for the outage was the unfortunate result of a small change to the production database, run manually by one of our coders. While the change was supposed to be a minor addition, it unforeseeably caused the database to become temporarily unavailable, effectively preventing our web services from working.

The nature of the issue left the system in a locked state which required manual intervention by our operations team. Rest assured that data integrity was never at risk, and that we did not have to restore any data from our backups.

We are sorry for any inconvenience caused by this outage. If you have any further questions regarding this incident, please do not hesitate to contact our customer support.

Lyyti Operations Team

Posted Oct 04, 2019 - 16:30 EEST

Resolved

This incident has been resolved. We will supplement a post-mortem report detailing the reason for the outage.

Posted Oct 04, 2019 - 14:59 EEST

Monitoring

Operation has been restored and everything is working normally as of 11:27 UTC. We will keep monitoring the system and research the underlying reason for the outage.

Posted Oct 04, 2019 - 14:28 EEST

Investigating

We are currently investigating a service outage starting ten minutes ago at 11:11 UTC. We will report back.

Posted Oct 04, 2019 - 14:22 EEST

This incident affected: Lyyti Portal, Lyyti registration, Messaging, API & Integrations, Lyyti Android application, and Lyyti iOS application.