Slow performance of Lyyti services

Incident Report for Lyyti

Postmortem

At the beginning of week 35, we observed slower than usual performance for our web services. We started investigating and soon determed the root cause of the issue to be twofold; increasing usage of our services, in combination with the ever growing amount of data in our database, caused the system to be insufficient in keeping up with the demand.

For all of last week, our development and operations teams have been working hard to implement multiple improvements in order to increase the response time and overall performance of our web services. Following is a list of some mitigating actions that have been implemented so far:

Code changes have been implemented in order to optimize some database queries that were found to be suboptimal. The same processes are performed as before, but some in a slightly faster manner.
Unnecessary data has been cleaned up and/or removed from the database. By decreasing the overall size of the database, resources are effectively freed to be used for the core system functionality.
Horizontal scalability has been deployed for the database layer; multiple servers now share the workload, and system performance is less likely to be affected by individual clients or users.
We have deployed new performance monitoring software that provides us with fine-grained analysis of performance and system workloads. These tools let our operations team not only monitor, but also proactively detect and mitigate anomalies before they become problems in the future.

We are confident that these actions will help us avoid similar incidents in the future. If you have any further questions regarding this incident, do not hesitate to contact our customer support.

Lyyti Operations Team

Posted Sep 03, 2019 - 15:07 EEST

Resolved

The Lyyti services have been performing normally all day today, and we will therefore close this incident now. As always, our operations team will of course keep monitoring our services and make sure to take any action as deemed necessary. We will also report back with a post-mortem report of the incident after the weekend.

Posted Aug 30, 2019 - 16:55 EEST

Monitoring

We have managed to return the service speed back to normal. We are still monitoring the situation but during today all operations are back to normal. We will later post a post mortem on the issue. We thank you for your understanding.

Posted Aug 30, 2019 - 12:07 EEST

Update

We are working hard on multiple improvements to fix the speed issue. We will post further updates as we progress with our work. We apologise for the issue and thank you for your understanding.

Posted Aug 29, 2019 - 09:59 EEST

Identified

Despite the performance increasing action, some of the issue still persists. We are working to fix it. Sorry for the inconvenience and thank you for your understanding.

Posted Aug 28, 2019 - 08:55 EEST

Monitoring

Some performance increasing action has been taken and some is still in the works. Currently Lyyti is performing normally, but we will of course keep monitoring the situation and keep you updated as needed.

Posted Aug 27, 2019 - 18:26 EEST

Investigating

Some of our services are currently working with slower than usual performance. We are currently implementing some changes to increase database performance. We are also continuously monitoring the situation, and will keep you informed as the situation progresses. Sorry for the inconvenience and thank you for your understanding.

Posted Aug 27, 2019 - 13:58 EEST

This incident affected: Lyyti Portal, Lyyti registration, Messaging, API & Integrations, Lyyti Android application, and Lyyti iOS application.