Resolved -
Received confirmation from Microsoft that their storage servers have been fully restored. It has not been 30 minutes and there have been no abnormalities found with our CloudCore infrastructure. Marking incident as resolved.
Mar 22, 22:06 CDT
Monitoring -
Azure storage servers have stabilized for the most part. VMs are mostly restored and device counts are back to near normal levels. Will continue to monitor and track open incident with Microsoft.
Mar 22, 20:49 CDT
Update -
We are working with Microsoft support now and confirmed that there is a storage issue at Azure's East US datacenter. This is triggering some VMs to reboot. OMC services will be flapping until Azure storage server issues are resolved.
This is only impacting the OMC portion of CloudCore. EPC and BOSS servers which are located at a different datacenter are not impacted.
Mar 22, 20:10 CDT
Identified -
One of the VMs experienced an abnormal reboot which is being reviewed. Upon reboot, the database services did not properly start which lead to the OMC outage. OMC services are in the process of being restored now and we are monitoring as devices come back online.
Mar 22, 19:34 CDT
Investigating -
OMC portion of CloudCore is partially down which is causing OMC data to not load correctly when viewed from CloudCore. Devices are also experiencing TR069 connectivity issues. Cloud OMC problems are currently under investigation.
Mar 22, 18:30 CDT