Investigating Database connection issues with host

Incident Report for ClickFunnels Classic

Resolved

This issue has been fully resolved.

Posted Nov 16, 2015 - 18:00 MST

Monitoring

User pages have been back online for a while now as we monitor the situation.

Funnel Editing in the app is coming back online now.

Background job processing will be delayed while we catch it up this evening.

Posted Nov 16, 2015 - 15:39 MST

Update

User pages are coming back online now - editing pages will remain down for a little longer while we confirm changes.

Status update video from Russell shared: https://www.facebook.com/groups/ClickFunnels/permalink/715182801914764/

Posted Nov 16, 2015 - 14:51 MST

Update

In progress still based on current progress, ETA estimate is 4:30PM EST

Posted Nov 16, 2015 - 12:27 MST

Update

We're getting close, emergency move to AWS Aurora Cluster is progressing much faster then restoration on original database cluster. We expect to finish migration to new AWS cluster fairly soon, much sooner than when original cluster completes restore.

Posted Nov 16, 2015 - 10:52 MST

Update

Our previous database cluster is still being restored.. We're also taking this time (while this outage is occurring & without any additional downtime) to migrate to an AWS Aurora cluster that's running AWS's top tier instances being 5x+ more powerful and virtually unlimited storage upside to avoid downtime needed for Database upgrades in the future.

Posted Nov 16, 2015 - 08:18 MST

Identified

Our database host has identified a problem across all servers and is working to get things restored. We'll update here as soon as things are back to normal.

Posted Nov 16, 2015 - 05:49 MST

Investigating

We are currently investigating this issue.

Posted Nov 16, 2015 - 05:07 MST