After monitoring the OMC for a few hours, no further issues have been discovered and the domain proxy service has been operating in a normal state. Marking the incident as resolved.
In summary, the Google SAS server was operating as expected and the outage was due to some maintenance made on the test servers last night which were mistakenly replicated to our live servers.
Posted Feb 27, 2024 - 15:39 CST
Monitoring
Root cause has been identified. Last night, our development team was prepping the test environment for the new CloudCore update we will be announcing soon. While updating the test environment, our development team mistakenly triggered grant relinquishes for some CBSDs on the commercial environment. This ultimately caused the heartbeat issues that were occurring this morning. We will be evaluating internally on how exactly this happened and will prevent this from occurring in the future.
As for now, any remaining heartbeats which receive 103 responseCode will automatically trigger a deregistration and reregistration. Will keep this incident open and continue to monitor to ensure all CBSDs are operating in a normal state.
Posted Feb 27, 2024 - 11:51 CST
Identified
The heartbeat errors due to 103 response code are due to the grant IDs being invalid. However, the grant IDs were never changed and the response codes are only found after the CPAS event this morning. We are still working with Google to understand the root cause.
The workaround for the time being would be to deregister and reregister the CBSDs. To manually perform this task please refer to the following steps:
1. Login to CloudCore 2. Navigate to Advance>SAS page 3. Click the checkbox next to each impacted CBSD 4. Click the "SAS OFF" button 5. Wait for CBSDs to indicate the SAS is disabled 6. Click the "SAS ON" button
Posted Feb 27, 2024 - 10:02 CST
Investigating
We are currently investigating an issue with some heartbeat messages erroring out with Google SAS with a 103 INVALID VALUE code. Due to the heartbeat errors, the impacted eNBs are disabling RF once the transmitExpire time triggers. This will impact all users who have CBRS equipment and are using Google SAS.