24/7/365 Sales & Support 1.866.257.8455 Login

System Status

System Status: Operational with Exceptions

07/05/2018 02:17 PM   |   0
Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Impaired Virtual Machines Restarting or otherwise troubleshooting issues with individual VMs.
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Normal SAN stcl01-node01 is back online. Root cause has been tracked back to bug in Ontap.
Normal L3 Switches All switches operating normally
Normal Firewall Clusters All firewally clusters operating normally
Normal Internet Connectivity Internet connectivity is operating normally

Notes: Admin staff is communicating with customers who have opened up cases. Website still down; waiting on developers to determine cause. NPM shows all VMs up. Some customers still reporting issues with their individual VMs. Ontap upgrade scheduled for Sunday morning.

System Status: Operational with Exceptions

07/05/2018 12:54 PM   |   0
Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Impaired Virtual Machines VMs running on storage controller stcl01-node01 are being restarted.
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Impaired SAN stcl01-node01 is back online. Logs being reviewed to determine root cause.
Normal L3 Switches All switches operating normally
Normal Firewall Clusters All firewally clusters operating normally
Normal Internet Connectivity Internet connectivity is operating normally

Notes: Operations VMs have been returned to normal operation. Engineers are using network performance monitor to determine which servers are still down. 80% of the VMs origninally reporting down are now responding to NPM. We are finding that, while VMs are showing up in NPM, they still need to be rebooted to ensure proper operation. Efforts to reboot all affected machines is ongoing.

System Status: Operational with Exceptions

07/05/2018 12:19 PM   |   0
Status System Notes
Normal vCenter vCenter is operating normally
Impaired vCloud Director vCloud Director is rebooting.
Impaired Virtual Machines VMs running on storage controller stcl01-node01 are being restarted.
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Impaired SAN stcl01-node01 is back online. Logs being reviewed to determine root cause.
Normal L3 Switches All switches operating normally
Normal Firewall Clusters All firewally clusters operating normally
Normal Internet Connectivity Internet connectivity is operating normally

Notes: A team is currently assigned to restoring service to Operations VMs to ensure system stability. A second team is assigned to restoring individual customer VMs.

System Status: Operational with Exceptions

07/05/2018 12:00 PM   |   0
Status System Notes
Normal vCenter vCenter is operating normally
Impaired vCloud Director vCloud Director is offline.
Impaired Virtual Machines VMs running on storage controller stcl01-node01 are offline or paused.
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Impaired SAN Los Angeles storage cluster stcl01 has experienced a kernel panic.
Normal L3 Switches All switches operating normally
Normal Firewall Clusters All firewally clusters operating normally
Normal Internet Connectivity Internet connectivity is operating normally

Notes: In Los Angeles, at approximately 11:54am PDT, one of our storage clusters has experienced an exceptional failure. stcl01-node01 reports a kernel panic, and the secondary node did not take over as designed. We are working to bring the storage components and affected VMs back online.

System Status: Operational

10/03/2016 01:00 PM   |   0

Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Normal Virtual Machines All VMs are operating normally
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Normal SAN All SAN components operating normally
Normal L3 Switches All switches operating normally
Normal Firewall Clusters All firewally clusters operating normally
Normal Internet Connectivity Internet connectivity is operating normally

Notes: All customer traffic in Boston has normalized. We will work with JTAC to determine root cause.

System Status: Operational

10/03/2016 12:36 PM   |   0

Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Normal Virtual Machines All VMs are operating normally
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Normal SAN All SAN components operating normally
Normal L3 Switches All switches operating normally
Normal Firewall Clusters All firewall clusters operating normally
Normal Internet Connectivity Internet connectivity is operating normally

Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. STRM shows a single customer VM heavily syn flooding. Shut down network interface and traffic normalizing. I have asked JTAC to explain how this is possible, since we have syn flood screens on this interface.

System Status: Operational with Exceptions

10/03/2016 12:30 PM   |   0

Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Normal Virtual Machines All VMs are operating normally
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Normal SAN All SAN components operating normally
Normal L3 Switches All switches operating normally
Impaired Firewall Clusters Connectivity issues through Boston firewall cluster
Normal Internet Connectivity Internet connectivity is operating normally

Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. JTAC is looking through logs and equipment status.

System Status: Operational with Exceptions

10/03/2016 11:59 AM   |   0

Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Normal Virtual Machines All VMs are operating normally
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Normal SAN All SAN components operating normally
Normal L3 Switches All switches operating normally
Impaired Firewall Clusters Connectivity issues through Boston firewall cluster
Normal Internet Connectivity Internet connectivity is operating normally

Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. The packet loss returned after approximately an hour. Performaing a simultaneous reboot and opening a case with JTAC to get to root cause as quickly as possible.

System Status: Operational with Exceptions

10/03/2016 10:35 AM   |   0

Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Normal Virtual Machines All VMs are operating normally
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Normal SAN All SAN components operating normally
Normal L3 Switches All switches operating normally
Impaired Firewall Clusters Connectivity issues through Boston firewall cluster
Normal Internet Connectivity Internet connectivity is operating normally

Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. Performed cluster failover. The issue appearst to have normalized. We are monitoring.

System Status: Operational with Exceptions

10/03/2016 10:15 AM   |   0

Status System Notes
Normal vCenter vCenter is operating normally
Normal vCloud Director vCloud Director is operating normally.
Normal Virtual Machines All VMs are operating normally
Normal VMware Hosts All hosts are operating normally
Normal VMware Clusters All clusters operating normally
Normal SAN All SAN components operating normally
Normal L3 Switches All switches operating normally
Impaired Firewall Clusters Connectivity issues through Boston firewall cluster
Normal Internet Connectivity Internet connectivity is operating normally

Notes: Extreme packet loss observed in Boston. Looking into the cause.