GPU & Node Health
GPU temperature, power, clocks and throttling reasons, ECC and XID errors, and utilization via NVIDIA DCGM exporter. Node CPU, memory, disk, and NIC state alongside it.
AI Cluster Monitoring · New
AI cluster monitoring.
Catch the bad link before it stalls the job.
A single flapping InfiniBand port or a GPU throwing XID errors can slow an entire distributed training job, and it rarely shows up as an outage. We watch GPU health and the compute fabric continuously, tune alerts to your cluster’s real baseline, and route pages to a senior engineer who has run production InfiniBand fabrics at hyperscale.
Experience
Run at hyperscale before we offered it to you.
This service comes from hands-on production operations of AI research infrastructure, across both the InfiniBand compute fabric and the Ethernet and storage networks around it.
Compute fabric
~2,400
InfiniBand-attached nodes operated in production.
IB switches
24
8 spine, 16 leaf, under both UFM and OpenSM control planes.
Ethernet
~560
Leaf/spine and edge switches around the compute and storage tiers.
Footprint
4
Sites, ~5,600 inventoried devices.
Prior-employer engagement; the client is not named. Figures describe the environment operated, not an Ambio Edge customer.
What Gets Monitored
GPUs, nodes, and the fabric between them.
Training performance depends on the slowest link in the collective. Coverage spans the whole path, so a slowdown can be traced to a component instead of guessed at.
InfiniBand Fabric
Symbol and link-integrity errors, link-downs, port resets, and ports that trained below expected width or speed. Subnet manager health on UFM or OpenSM. Alerts on rate of change, not lifetime counters.
RoCE / Ethernet Fabric
Leaf/spine link errors and drops, PFC pause and ECN activity on lossless classes, LACP and uplink state. The congestion signals that quietly stretch step times.
Storage Network Path
Error and throughput telemetry on the links between compute and storage, so a data-loading stall is traced to a port rather than blamed on the model.
Topology Drift
Discovered fabric topology compared against the expected cabling plan. A moved cable or a missing link is flagged before it becomes a mystery slowdown.
Alerting & Routing
Alert rules tuned to your cluster’s baseline, with severity tied to job impact. Critical pages go to a senior engineer with the affected ports and nodes already identified.
Your Engineer
One engineer. Fabric-level context.
Not a team of rotating technicians. Not a ticket queue. One named senior engineer who knows your environment, your compliance requirements, and your business context — from assessment through ongoing operations.
JJ
Jeff Johnson
Principal Network Architect
The person who designs your network is the person who maintains it. No handoffs. No abstraction. No loss of context when something breaks.
Background: Operated production InfiniBand and Ethernet fabrics for hyperscale AI research infrastructure: ~2,400 IB-attached nodes, 24 IB switches, and ~560 Ethernet switches across 4 sites.
Cisco Certified
Fortinet Certified
CompTIA Certified
Fortinet Engage Partner
20+ Yrs
Network-Only
Technology Partners
Fortinet partner. Vendor-neutral advice.
We are a Fortinet Engage Advocate partner and supply FortiGate, FortiSwitch and FortiAP. We also monitor and manage the Cisco, Aruba and UniFi gear you already run. You don’t have to rip anything out to work with us.
Scope
Monitoring and alerting. Clearly bounded.
✓ Included
- Continuous GPU, node, and fabric telemetry
- Alert rules and dashboards tuned to your cluster
- Senior-engineer alert routing with a written diagnosis
- Works alongside your existing Prometheus / Grafana stack
× Not Included
- Fabric design, cabling, or hardware procurement
- Job scheduler, Kubernetes, or MLOps administration
- Model training or tuning work
- Hands-on remediation inside your cluster unless contracted
FAQ
Common AI cluster monitoring questions.
What does pricing look like?
AI cluster monitoring is scoped per environment, not sold off the per-site card. Node count, fabric type (InfiniBand, RoCE, or both), and alert-routing hours drive the price. You get a written quote after the assessment call.
Do you design or build the cluster fabric?
No. This service is monitoring and alerting only. You or your integrator own the architecture; we watch it, catch degradation early, and hand your team a specific diagnosis: which port, which node, since when.
How does telemetry leave our environment?
Through the Ambio edge connector: outbound-only encrypted tunnels from a small VM or appliance on your side. No inbound ports are opened at your site. Collectors can also read from your existing Prometheus, DCGM exporter, or UFM API instead of polling devices directly.
We already run Prometheus and Grafana. Do you replace it?
No. If your cluster already exports DCGM and node metrics, we build on that: add fabric telemetry, write alert rules tuned to your baseline, and route pages to an engineer. The dashboards and rules stay yours.
Start Here
Tell us about the cluster.
We’ll tell you what to watch.
One call to walk through your node count, fabric, and current tooling. You get a written monitoring plan and quote whether or not you continue with us.
What Happens Next
Response from a senior engineer within 1 business day.
A direct conversation — no sales team, no runaround.
A written monitoring plan for your cluster.
Clear next steps if we are a fit — no pressure if we are not.