Received confirmation from Microsoft that their storage servers have been fully restored. It has not been 30 minutes and there have been no abnormalities found with our CloudCore infrastructure. Marking incident as resolved.
Posted Mar 22, 2023 - 22:06 CDT
Monitoring
Azure storage servers have stabilized for the most part. VMs are mostly restored and device counts are back to near normal levels. Will continue to monitor and track open incident with Microsoft.
Posted Mar 22, 2023 - 20:49 CDT
Update
We are working with Microsoft support now and confirmed that there is a storage issue at Azure's East US datacenter. This is triggering some VMs to reboot. OMC services will be flapping until Azure storage server issues are resolved.
This is only impacting the OMC portion of CloudCore. EPC and BOSS servers which are located at a different datacenter are not impacted.
Posted Mar 22, 2023 - 20:10 CDT
Identified
One of the VMs experienced an abnormal reboot which is being reviewed. Upon reboot, the database services did not properly start which lead to the OMC outage. OMC services are in the process of being restored now and we are monitoring as devices come back online.
Posted Mar 22, 2023 - 19:34 CDT
Investigating
OMC portion of CloudCore is partially down which is causing OMC data to not load correctly when viewed from CloudCore. Devices are also experiencing TR069 connectivity issues. Cloud OMC problems are currently under investigation.