AWS (Server Farm) Issue
Incident Report for LivePOS
Postmortem

Dear Customers,

On Monday at approximately 2pm PST LivePOS have experienced a partial outage that effected POSs and created red logos.

As a former store owner I know how these outages can impact store operations and create needles stress. Please accept my sincere apologies.

As the head of IT here at LivePOS we will continue to work with our vendors and Data Center facilities to minimize these outages.

Please find a technical incident report below. Have a great holiday and thank you for choosing LivePOS.

David Miller, Head of IT.

Technical incident report:

At approximately 2pm PST our support center started to receive calls from customer reporting red logos and slow performance. Our IT department started to investigate the issue immediately. At 2:17 it was determined that requests from the POSs (to push sales etc) are degraded due to intermediate database connectivity on the Amazon Web Services (Data Center) Side. Our IT department reached out to AWS (Amazon Web Services) and reported the problem. At 3:12 AWS confirmed that one of their routers is malfunctioning and effecting the communication between the LivePOS servers and the RDS database.

At 3:55 AWS confirmed that it resolved the problem and network operations should be back to normal within 30-40 minutes. At approximately 5:45PM reports started to come in that red logos are disappearing and POS sync is back to normal. It took another 10-15 minutes for the AWS network to be back at 100%, most likely because of all the traffic that was held “at the gate” and was now trying to “push in” after the issue was resolved. During this time updates and status reports where posted on our LivePOS Status page and voice recording informed our customers of the issue. Luckily, this issue was resolved by AWS within a few hours, which eliminated the need to switch the entire LivePOS to our backup Datacenter. In addition, the LivePOS Offline feature provided merchants with the ability to continue ringing up customers while the issue is being resolved.

LivePOS Team

Posted Dec 16, 2015 - 09:50 PST

Resolved
This issue is now resolved. AWS (Data Center) support rebooted some hardware on their end and the issue was resolved. having said that, they are still not sure what happen and will continue investigating and give us updated throughout the night. We apologies for the inconvenience and thank you for your patience.
Posted Dec 14, 2015 - 19:13 PST
Investigating
We are investigating an issue where customers are reporting Red Logos.
Posted Dec 14, 2015 - 16:30 PST