CH-Jordan

At approximately 4PM CT yesterday (5/4/12), Cartika engineers were again alerted to an issue affecting our Dallas distribution and downstream access switching equipment. Immediately upon alert, Juniper JTAC Engineers were brought on to assist with our already ongoing case. During the next two hours and twenty minutes, many customers may have seen intermittent connectivity and performance as ongoing troubleshooting ensued. While inspecting all access switches, engineers found abnormal STP behaviour from one of our downstream access switches. All access switches on our network are connected in full mesh to the upstream distribution gear, and we make use of STP to block the redundant path until needed, thus preventing a L2 loop. However, in the case of this access switch, it was temporarily unblocking the redundant path, causing a subsequent broadcast storm to ensue. Meanwhile, upstream at the distribution layer, all access switches and paths reported healthy to engineers.

After diagnosing this issue, we swapped this access switch and believe the issue to be resolved. Many services that were dependent on network based storage (cloud) were impacted the hardest, although we worked as fast as possible to restore service once we rectified the aforementioned network issue. We are continuing to investigate the upstream traffic flows that triggered this condition, the bug that allowed it to propagate through our distribution fabric without being properly blocked, and further measures that can be taken to ensure that this issue does not reoccur with the assistance of JTAC.

Thank you for your continued business and please let us know if you have any questions.

web16 is fully booted, online, and stable at this time.

The failing memory chip has been replaced and we are powering up web16 at this time. We will update you again once web16 is completely online and stable.

This maintenance is getting underway and we will update once complete.

This maintenance has been completed successfully.

Thank you for your continued business and as always, if you have any comments or questions, please do not hesitate to contact us.