OMC Database Sync Issue
Incident Report for Baicells Technologies
Postmortem

Cloud OMC & BOSS 7.1 Upgrade has been Postponed

The Cloud OMC & BOSS 7.1 upgrade which was originally planned for August 4th has been postponed. While prepping for the upgrade today, we had to prematurely stop the database synchronization due to the sync process taking longer than expected. Upon stopping the database synchronization, we found additional issues with eNBs not fully syncing correctly with the OMC which lead to some eNBs with SAS enabled dropping to an inactive state.

We would like to apologize about the inconvenience and loss of service that this cost our customers and their clients. This upgrade will be postponed until the syncing issues have been fully identified and addressed and a new upgrade date will be announced when ready through all our channels.

Posted Aug 03, 2021 - 18:02 CDT

Resolved
In preparation for the OMC upgrade on August 4th, 2021, the OMC database servers were synchronized 24 hours prior to the OMC upgrade (August 3rd, 2021 at Midnight PT), which was expected to last for 5 hours, however, during the database sync, the sync process was taking longer than the expected window and was stopped. Users have also reported that the communication between eNB and CPE stopped, and user traffic was impacted. We are currently investigating to see what could have caused the issue and will update all channels as information becomes available.

UPDATE: We are still investigating internally. We did however find that we had to manually sync some eNBs that were still Inactive. If anyone is still experiencing issues with the OMC connection or their eNBs being Inactive, the issue may be recovered by performing a manual sync on the eNB. To perform the manual synchronization, navigate to OMC>Monitor page and click the options (ellipsis) button next to the respective eNB SN and then click Actions>Synchronize. (edited)
Posted Aug 03, 2021 - 11:38 CDT
This incident affected: Cloud OMC.