After monitoring the OMC servers throughout the night and this morning, we can confirm that the incident has been fully resolved. The reporting issues were caused by high latency database queries which lead to a lot of connection timeouts. Configuration changes and optimizations were ultimately made to bring response times to normal levels.
A full internal investigation is under way to determine why no issues were discovered during our standard testing process.
Posted Jan 11, 2019 - 12:53 CST
Monitoring
A fix was implemented a couple hours ago and we are continuing to monitor the results. The TR-069 services are running normal now and all devices have been slowly recovering their connection with the OMC. Will report further details once we fully confirm the issue is fully resolved.
Posted Jan 10, 2019 - 21:00 CST
Investigating
Following the OMC upgrade this morning, we have found that a large number of eNodeBs and CPEs are reporting offline. We are still investigating the issue and we ask that you ignore any reporting errors from the OMC until the issue is fully resolved. Another update will be provided to confirm this.
The current OMC reporting issues do not have any impact on network connectivity. For any HaloB-enabled eNBs, the exception to this is if you are trying to attach a new subscriber. If the HaloB eNB is disconnected from the OMC, then it will not be able to download new subscriber profiles. To resolve this issue, the workaround is to log into the eNB webGUI, disable HaloB, and reboot. After reboot, the eNB will be using Cloud EPC and all active IMSIs will be able to attach without issue.
We appreciate everyone's patience during this time.