[HRN] RG-US-MI6 Random Reboots on 31 Dec 2025 18:13:11 (UTC-05:00)
Resolved on 2 Jan 2026 11:07:28 (UTC-05:00)
Server Stable
We are now closing this incident because the server remains stable after swapping the motherboard. Please enter a ticket if you have lingering issues post motherboard swap.
Monitoring on 1 Jan 2026 21:29:15 (UTC-05:00)
Repair Complete
The motherboard replacement was completed at 5:10 PM ET, and all services are now online and operating normally. We will continue to closely monitor the server over the next several hours to ensure ongoing stability.
Identified on 31 Dec 2025 21:44:31 (UTC-05:00)
We have identified a motherboard failure on MI6, which is causing service disruption for affected customers. A replacement motherboard has been sourced, and on-site repairs are scheduled to begin around 5:00 PM ET on January 1, 2026.
Due to New Year’s Day and road conditions, replacement timelines are limited. While data loss is unlikely, there is a small risk inherent to hardware failures. Customers with recent backups are encouraged to ensure they are available.
Further updates will be posted as repair work progresses.
Hardware Failure
We have identified a motherboard failure on MI6, which is causing service disruption for affected customers. A replacement motherboard has been sourced, and on-site repairs are scheduled to begin around 5:00 PM ET on January 1, 2026.
Due to New Year’s Day and road conditions, replacement timelines are limited. While data loss is unlikely, there is a small risk inherent to hardware failures. Customers with recent backups are encouraged to ensure they are available.
Further updates will be posted as repair work progresses.
Investigating on 31 Dec 2025 18:13:11 (UTC-05:00)
We are aware of an issue with MI6 where the machine keeps going unreachable on/off since 16:15 ET. Upon some investigation, we found that the server is sporadically hard rebooting.
The root cause is still being investigated.
We are aware of an issue with MI6 where the machine keeps going unreachable on/off since 16:15 ET. Upon some investigation, we found that the server is sporadically hard rebooting.
The root cause is still being investigated.
[HRN] November 9, 2025 Network Spine Replacement on 29 Oct 2025 09:56:05 (UTC-05:00)
Update on 9 Nov 2025 14:13:16 (UTC-05:00)
Monitoring and testing complete.
Monitoring and testing complete.
Update on 9 Nov 2025 11:54:17 (UTC-05:00)
The spine switch has been swapped. For the remainder of the maintenance window, we will be monitoring stability and testing various things.
The spine switch has been swapped. For the remainder of the maintenance window, we will be monitoring stability and testing various things.
Scheduled on 29 Oct 2025 09:56:05 (UTC-05:00)
We’ll be performing a critical network maintenance on Sunday, November 9, 2025, starting at 11:30 AM Eastern, with an expected duration of approximately 1.5 hours.
During this window, you will experience periods of network downtime as we replace our existing spine switch with a higher-capacity model. This upgrade is necessary to accommodate our continued growth and to ensure long-term network stability and performance.
- Greatly increase core capacity and throughput
- Improve routing efficiency and BGP convergence times
- Enhance redundancy across all customer services
- Affected services: ALL SERVICES
- Customer action: None required — all systems will automatically recover once maintenance is complete
- 11:30–12:30 ET: Spine replacement and configuration validation
- 12:30–13:00 ET: Post-maintenance verification and monitoring
We know this follows closely after our last network change, and we don’t take that lightly. This upgrade directly addresses the hardware limitation identified from the previous work and ensures we won’t need another major window for this part of the network in the foreseeable future.
Note: We understand that maintenance windows can be inconvenient. We schedule them carefully for times when our small core team is available on-site to perform and monitor changes directly. Each maintenance is planned with safety, reliability, and long-term stability in mind. We truly appreciate your understanding.
We schedule these changes proactively to minimize unplanned downtime in the future and to keep our network reliable for everyone.
If you notice any irregularities after 13:30 ET, please open a ticket with relevant details (IP, timestamps, traceroute if possible) so we can investigate quickly.
Thank you for your continued trust as we expand and strengthen our network.
We’ll be performing a critical network maintenance on Sunday, November 9, 2025, starting at 11:30 AM Eastern, with an expected duration of approximately 1.5 hours.
During this window, you will experience periods of network downtime as we replace our existing spine switch with a higher-capacity model. This upgrade is necessary to accommodate our continued growth and to ensure long-term network stability and performance.
Why this is happening:
Our current spine switch, installed during the last maintenance, has proven insufficient for the level of throughput and routing scale we’re now reaching. Replacing it allows us to:- Greatly increase core capacity and throughput
- Improve routing efficiency and BGP convergence times
- Enhance redundancy across all customer services
What to expect:
- Impact: Intermittent downtime during the transition and validation process- Affected services: ALL SERVICES
- Customer action: None required — all systems will automatically recover once maintenance is complete
Estimated timeline:
- 11:15–11:30 ET: Preparation- 11:30–12:30 ET: Spine replacement and configuration validation
- 12:30–13:00 ET: Post-maintenance verification and monitoring
We know this follows closely after our last network change, and we don’t take that lightly. This upgrade directly addresses the hardware limitation identified from the previous work and ensures we won’t need another major window for this part of the network in the foreseeable future.
Note: We understand that maintenance windows can be inconvenient. We schedule them carefully for times when our small core team is available on-site to perform and monitor changes directly. Each maintenance is planned with safety, reliability, and long-term stability in mind. We truly appreciate your understanding.
We schedule these changes proactively to minimize unplanned downtime in the future and to keep our network reliable for everyone.
If you notice any irregularities after 13:30 ET, please open a ticket with relevant details (IP, timestamps, traceroute if possible) so we can investigate quickly.
Thank you for your continued trust as we expand and strengthen our network.