Duration: 10 minutes

Affected Components: app.pindo.io, api.pindo.io

Oct 13, 14:20:00 GMT+0
Resolved - This incident has been resolved..

Oct 13, 14:16:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Oct 13, 14:15:00 GMT+0
Identified - Airtel SMPP was not responding caused by VPN. We shifted to secondary provider as mitigation while fixing the issue.

Oct 13, 14:10:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 10 minutes

Affected Components: api.pindo.io

May 2, 07:04:00 GMT+0
Resolved - This incident has been resolved..

May 2, 07:00:00 GMT+0
Monitoring - We have revived all the workers and are currently monitoring the result..

May 2, 06:59:00 GMT+0
Identified - SMS delivery delays are caused by a couple of workers which were dead and didn't auto-heal. We have now fixed it..

May 2, 06:54:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 35 minutes

Affected Components: api.pindo.io

Mar 18, 17:48:00 GMT+0
Resolved - This incident has been resolved..

Mar 18, 17:18:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 18, 17:14:00 GMT+0
Identified - We have identified the issue with task worker servers failing and started the manual fallback to reinstate them..

Mar 18, 17:13:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 12 hours and 4 minutes

Feb 4, 06:53:00 GMT+0
Resolved - This incident has been resolved..

Feb 4, 06:52:00 GMT+0
Monitoring - Failed workers auto healing mechanism completed and delays are over..

Feb 4, 06:51:00 GMT+0
Identified - We had a temporary degradation of SMS delivery due to some worker failure which automatically restarted..

Feb 4, 06:49:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 17 minutes

Affected Components: api.pindo.io

Jan 30, 07:40:00 GMT+0
Resolved - After many test we can conclusively declare the issue resolved..

Jan 30, 07:32:00 GMT+0
Monitoring - We have fixed the issue with the background worker. We are monitoring it to make sure it is working as intended.

Jan 30, 07:28:00 GMT+0
Identified - The failure is caused by background worker responsible to schedule and send SMS.

Jan 30, 07:23:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 8 minutes

Affected Components: api.pindo.io, app.pindo.io, pindo.io, smpp.pindo.io

Jan 26, 13:18:50 GMT+0
Resolved - This incident has been resolved. .

Jan 26, 12:28:00 GMT+0
Monitoring - We have resolved the issue with auto-renew by forcing the payment for the domain. We have engaged the domain provider to fix the auto-renew issue so that we don't find ourselves in that position again.

Jan 26, 12:20:00 GMT+0
Identified - We found-out that our domain provider though they have auto-renew it didn't take effect and flagged our domain expired.

Jan 26, 12:11:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 23 minutes

Affected Components: api.pindo.io, app.pindo.io

Nov 10, 08:18:00 GMT+0
Resolved - This incident has been resolved. We are currently using the secondary channel. We will shift back to primary once we have full assurance on its performance.

Nov 10, 08:10:00 GMT+0
Monitoring - We have shifted to the secondary SMS channel and are monitoring.

Nov 10, 08:03:00 GMT+0
Identified - Our primary channel is affected and we are shifting to secondary channel to avoid continued details while we are investigating.

Nov 10, 07:55:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 39 minutes

Affected Components: api.pindo.io, app.pindo.io

Oct 29, 11:30:00 GMT+0
Resolved - This incident has been resolved. The deployed solution is working effectively..

Oct 29, 11:02:00 GMT+0
Monitoring - Deployed a fix for task manager and tested SMS delivery.

Oct 29, 10:55:00 GMT+0
Identified - Identified that the SMPP was working well but the task manager was not processing well messages.

Oct 29, 10:51:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 13 hours and 1 minute

Affected Components: api.pindo.io, app.pindo.io

Oct 28, 08:29:00 GMT+0
Resolved - This incident has been resolved. We are now processing all rejected SMS to assure all messages are sent .

Oct 28, 07:59:00 GMT+0
Monitoring - The primary telco fixed the issue and we are shifting back to the primary now and monitor it..

Oct 28, 07:29:00 GMT+0
Identified - Switched to Secondary telco while engaging the primary to solve the issue on their end.

Oct 28, 07:28:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 12 hours and 6 minutes

Affected Components: api.pindo.io

Oct 2, 19:26:00 GMT+0
Resolved - This incident has been resolved. No more delays..

Oct 2, 19:25:00 GMT+0
Monitoring - We have fixed the issue and are monitoring status of SMS delivery.

Oct 2, 19:22:00 GMT+0
Identified - We had identified the miss-communication between task queue and the SMPP.

Oct 2, 07:20:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 20 minutes

Affected Components: api.pindo.io, pindo.io, app.pindo.io, smpp.pindo.io

Aug 21, 11:12:00 GMT+0
Resolved - This incident has been resolved..

Aug 21, 11:02:00 GMT+0
Monitoring - The resources have been increased and now are monitoring.

Aug 21, 10:56:12 GMT+0
Identified - Failed to auto scale servers in relation to SMS payload. We are manually increasing server resources.

Aug 21, 10:52:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 50 minutes

Affected Components: app.pindo.io

Aug 21, 10:52:00 GMT+0
Resolved - This incident has been resolved..

Aug 21, 10:29:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 21, 10:15:00 GMT+0
Identified - Some endpoints are not accessible. We are working on a fix.

Aug 21, 10:02:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 25 minutes

Affected Components: api.pindo.io, app.pindo.io

Jul 21, 10:08:00 GMT+0
Resolved - This incident has been resolved..

Jul 21, 10:03:00 GMT+0
Monitoring - Requested support for the telco which solved the issue. We have switched back to primary telco and now monitoring....

Jul 21, 09:51:00 GMT+0
Identified - The issue is related to one telco we have switched to secondary telco to avoid service interption.

Jul 21, 09:43:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 42 minutes

Affected Components: smpp.pindo.io, api.pindo.io, pindo.io, app.pindo.io

Jul 11, 07:09:00 GMT+0
Resolved - MTN fixed the issue with SMSC. We have reverted back to MTN as primary. The system is back to normal.

Jul 11, 06:30:00 GMT+0
Monitoring - The switch was smooth. We are currently, monitoring and SMS are being delivered.

Jul 11, 06:29:00 GMT+0
Identified - We have identified that MTN SMC was not working and directly engaged MTN for support. We have switched to the secondary provider to avoid any downtime..

Jul 11, 06:27:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 26 minutes

Affected Components: api.pindo.io

Jun 11, 07:40:00 GMT+0
Resolved - This incident has been resolved..

Jun 11, 07:38:00 GMT+0
Monitoring - A part of task manager malfunctioned and was not connecting directly to the carrier this created a queue. We have re-established the link and now queue is being processed..

Jun 11, 07:34:00 GMT+0
Identified - We found out that thought messages are being sent to carrier, they are not being delivered as expected..

Jun 11, 07:14:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 22 minutes

Affected Components: api.pindo.io

Apr 15, 14:04:00 GMT+0
Resolved - After continuous monitoring the issue is solved and going to upgrade the background server to avoid any potential of this happen again.

Apr 15, 13:58:00 GMT+0
Monitoring - We hare restarted the background task server and monitoring the effect while studying the cause.

Apr 15, 13:50:00 GMT+0
Identified - The background task seems to be overloaded. We are looking on the cause.

Apr 15, 13:42:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 24 minutes

Affected Components: api.pindo.io

Apr 6, 08:21:00 GMT+0
Resolved - After multiple tests. The system is back to normal.

Apr 6, 08:17:00 GMT+0
Monitoring - Deployed a fix and started monitoring. Now SMS are being sent and all pending message are now being handled.

Apr 6, 08:07:00 GMT+0
Identified - Found out that the background task was not responsive. Working onto fixing it.

Apr 6, 07:57:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 26 minutes

Affected Components: smpp.pindo.io

Mar 27, 11:19:00 GMT+0
Resolved - The queue is now close to zero and messages are being delivered. We are going to schedule a maintenance to upgrade some of our servers to handle well large bulk pipeline .

Mar 27, 11:17:00 GMT+0
Monitoring - Since bulk is separated from transactional connection. The issue subsided but with a long queue which we are monitoring.

Mar 27, 11:02:00 GMT+0
Identified - Message congestion on a long queue due to many messages sent at around the same time in bulk..

Mar 27, 10:53:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 13 hours

Affected Components: api.pindo.io

Mar 21, 20:21:00 GMT+0
Resolved - Tested if french characters are being rendered well and closed the issue.

Mar 21, 08:03:00 GMT+0
Monitoring - We have reverted back the changes to first notify customers of changes and have pricing per encoding introduced.

Mar 21, 07:32:00 GMT+0
Identified - The issue is associated with an attempt to adjust on telco charges over messages with ASCII-Extended characters.

Mar 21, 07:21:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 17 minutes

Affected Components: smpp.pindo.io

Mar 19, 18:08:00 GMT+0
Resolved - Conducted tests to assure the service is working well. Added mesures to avoid such issues in the future by conducting stress test close to midnight preferably in the weekends when there is less traffic..

Mar 19, 18:06:00 GMT+0
Monitoring - Stopped testing process. Restarted the server and fixed the issue. .

Mar 19, 18:03:00 GMT+0
Identified - SMPP server received multiple request while testing uptime mechanism which caused CPU and RAM spike and cause server shutdown..

Mar 19, 17:51:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 23 minutes

Affected Components: api.pindo.io

Feb 28, 07:12:00 GMT+0
Resolved - The primary provider have fixed the delivery issue and we switched back to it. This solves also the issue with SenderIDs—now all custom senderIDs are working as intended. We are continually engaging with the secondary provider to fix permanently their system to use senderIDs appropriately.

Feb 28, 07:03:00 GMT+0
Monitoring - Though message are being sent via secondary provider. We have noticed the issue with senderID which are being displayed as either "Airtel" or "ECP". We directly engaged the secondary provider to fix the issue..

Feb 28, 06:59:00 GMT+0
Identified - We have identified an issue with our primary provider and switched to the secondary provider to avoid any prolonged down-time.

Feb 28, 06:49:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 10 minutes

Affected Components: api.pindo.io

Feb 26, 07:22:00 GMT+0
Resolved - The issue is completely solved and we are back on the primary channel..

Feb 26, 07:20:00 GMT+0
Monitoring - Re-connected with primary channel SMPP and monitoring if delivery is working as expected.

Feb 26, 07:16:00 GMT+0
Identified - Identified the issue with the primary channel connection. Switched to secondary channel while fixing the underlying issue.

Feb 26, 07:12:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 48 minutes

Affected Components: api.pindo.io

Feb 22, 09:58:00 GMT+0
Resolved - After 15 minutes of observation we concluded that the issue is completely solved..

Feb 22, 09:45:00 GMT+0
Monitoring - After pinpointing the exact issue—server response failure—on our server we fixed it and started monitoring.

Feb 22, 09:41:00 GMT+0
Identified - We first thought that the issue is related to Telco and switched to secondary telco but that didn't solve the issue. After testing SMPP on both telcos we confirmed that the issue is on the servers end instead.

Feb 22, 09:10:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 4 minutes

Affected Components: api.pindo.io, app.pindo.io

Jan 6, 09:19:00 GMT+0
Resolved - Everything is working and back to normal.

Jan 6, 09:18:00 GMT+0
Monitoring - All server instances are not back to normal and currently monitoring them.

Jan 6, 09:16:00 GMT+0
Identified - Some servers where not reachable and took more time than usual to to auto-provision.

Jan 6, 09:15:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 10 minutes

Affected Components: api.pindo.io

Jan 3, 13:22:00 GMT+0
Resolved - Switched back to primary after MTN Rwanda solved their internal internet connectivity issue.

Jan 3, 13:20:00 GMT+0
Monitoring - Delivery of SMS using the secondary channel is working fine..

Jan 3, 13:15:00 GMT+0
Identified - MTN Rwanda general connectivity issue made their SMPP servers unavailable. We have initiated the switch to secondary provider to avoid downtime.

Jan 3, 13:12:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours

Affected Components: api.pindo.io, pindo.io, app.pindo.io, smpp.pindo.io

Nov 9, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 8, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 8, 23:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 12 hours and 5 minutes

Affected Components: api.pindo.io, app.pindo.io

Dec 6, 22:00:00 GMT+0
Completed - Maintenance has completed successfully..

Dec 6, 09:55:00 GMT+0
Identified - The previous retry priority algorithm as performant as it was. It had a key component of monitoring failed messages and due to backlog of retry occasionally we could be in a situation where there is a large set of messages due to retry and this could overwhelm our provide Delivery Report service and end up with a large number messages in an intermediate state. The new implementation handle the issue stated above and went beyond on having a specific tracking system to identify residual request for retry and discard them. This makes the retry priority better and resolve completely the potential to overwhelm the telco delivery report service in the future..

Duration: 2 hours

Affected Components: api.pindo.io, app.pindo.io, pindo.io

Aug 9, 00:00:00 GMT+0
Completed - Maintenance has completed successfully..

Aug 8, 23:00:00 GMT+0
Identified - Started database migration without interruption.

Aug 8, 22:00:00 GMT+0
Identified - Staring the deployment with database migration.

Duration: 5 days

Affected Components: api.pindo.io, app.pindo.io

Jul 30, 00:00:00 GMT+0
Completed - Maintenance has completed successfully..

Jul 29, 22:05:00 GMT+0
Identified - We are planning a major deployment which will affect the database.

Jul 29, 22:00:00 GMT+0
Identified - We are planning a major deployment which will affect the database.