PSI PULSE — Complete Operations Prototype v1.6 Platform API Expansion

GPU / MIG Operations Dashboard

Detailed Grafana-style visual monitoring combined with PSI PULSE L1 workflow and fleet context.

Live simulation

Aggregate

1,024 servers • 8,192 GPUs • 3 halls

GPU Engine Utilization

70.2%

fleet aggregate

Tensor Utilization

41.8%

fleet aggregate

GPU Memory Utilization

60.1%

fleet aggregate

GPU Memory Usage

1.41 PB

allocated framebuffer

Per GPU

DCGM real-time metrics

GPU Temperature °C

Power Usage W

GPU Energy Draw Total

Last 1h

1.26 MWh

Last 24h

28.7 MWh

Clock & Instance Type

SM Clock

1.76 GHz

Memory Clock

1.59 GHz

GPU Utilization

Tensor Utilization

Memory Utilization

Memory Temperature

NVLink Bandwidth

GPU Instance Type

B300 HGX
8,192 GPUs

Per MIG Profile

Memory, utilization and allocation

MIG Memory Usage Trend

MIG Memory Usage Heat Bars

MIG GPU Utilization

MIG Tensor Utilization

MIG Memory Utilization

MIG Allocation Table

Hostname	Hall / Rack	GPU ID	MIG Profile	GPU Instance	Namespace	Pod	GPU Util	Memory	Health
h4-r12-n031	H4 / R12	0	1g.24gb	13	ai-inference	model-a-7f9	81%	20.1 GB	Healthy
h4-r12-n031	H4 / R12	1	2g.48gb	3	ai-training	train-llm-02	94%	43.6 GB	Healthy
h3-r07-n018	H3 / R07	0	2g.48gb	4	vision-prod	vision-24	37%	31.2 GB	Thermal watch
h3-r08-n004	H3 / R08	3	1g.24gb	9	ai-inference	embed-03	0%	3.2 GB	XID detected

Error & Reliability

PCIe, XID and ECC

PCIe Retry / Replay Rate

XID Error

ECC Corrected / Uncorrected

L1 Operations Queue

Correlated by PSI PULSE

Active Alerts & Incidents

P2

Repeated XID 31 on four GPUs in Hall 3

Probable driver / firmware mismatch • Confidence 72% • INC-2472

03:11 SLA

P2

Leaf-07 BER trend threatens redundant path

42 nodes at risk • OEM case recommended

00:42 SLA

P3

GPU temperature deviation above rack baseline

H3-R07 • facility context required

05:28 SLA

Pulse Agent Guidance

Immediate L1 validation

Collect DCGM diagnostics and BMC events. Do not reset GPU until workload drain approval.

Preventive network action

Validate optics and BER counters on Leaf-07 and prepare OEM evidence bundle.

Capacity optimization

17 idle MIG instances are candidates for scheduler review, not automatic reassignment.

PSI PULSE COMMAND CENTER

Live Operations • NeutraDC Batam

00:00:00

Multi-Hall Fleet Status

Active Incident Queue

P2

XID cluster — Hall 3

4 GPUs

03:11

P2

Leaf-07 BER degradation

42 nodes

00:42

Fleet Availability

99.94%

1,018 of 1,024 servers healthy

GPU Capacity

8,176

Available GPUs

Pulse Agent Early Warning

BER trend accelerating

Inspect Leaf-07 optics within 24 hours.

Readiness

Open approvals2

Maintenance02:00–04:00

Audit integrityVerified

Onboarding & Provisioning

Real-lab inspired machine lifecycle using DHCP discovery, NetBox staging, MAAS commissioning and Redfish power control. PSI PULSE renders the governed state projected by VM100.

B300 lab reference

1. BMC LeaseKea detects BMC MAC/IP on OOB network.

2. DiscoveryLease event enters discovery pipeline.

3. NetBox StagedCanonical candidate enriched with rack/model/MAC context.

4. MAAS EnlistPXE/enlist creates machine record.

5. CommissionHardware inventory and Redfish power validation.

6. DeployApproved image and network profile.

7. ActiveNetBox/VM100 state promoted after evidence.

Discovered

16

Staged

16

Commissioning

4

Ready

12

Failed / blocked

0

Machine Queue Lab-derived sample

Machine	Rack/U	BMC MAC	OOB IP	Power	MAAS	PSI State
sim-node-01	rack-a01 / U1	02:b3:00:00:00:01	172.30.15.101	Redfish OK	Ready	Qualified
sim-node-02	rack-a01 / U11	02:b3:00:00:00:02	172.30.15.102	Redfish OK	Ready	Qualified
sim-node-03	rack-a01 / U21	02:b3:00:00:00:03	172.30.15.103	Redfish OK	Commissioning	Evidence pending
sim-node-04	rack-a01 / U31	02:b3:00:00:00:04	172.30.15.104	Redfish OK	Commissioning	Evidence pending

Selected Machine / Commissioning Evidence

Device typeHGX B300 (sim), 10U

Power typeRedfish / sushy-tools

PXE network10.10.30.0/24

OOB network172.30.15.0/24

Expected interfacesbmc / mgmt0 / pxe0 / data0

Evidence rule

Frontend never marks a node Active from UI state alone. VM100 must receive authoritative lifecycle/evidence before promotion.

Fleet & Assets

Canonical inventory, OEM models, firmware, contracts and telemetry readiness.

Servers

1,024

1,018 healthy

GPUs

8,192

16 observed

Switches

96

95 healthy

Telemetry Sources

4,286

22 degraded

Asset Inventory

Asset ID	Hall/Rack	OEM/Model	GPU	BMC	Firmware	Contract	Health
HGX-H4-R12-N031	H4/R12	OEM-A X9000-HGX	8×B300	Reachable	2026.06-A	24×7	Healthy
HGX-H3-R08-N004	H3/R08	OEM-B AI Rack	8×B300	Reachable	2026.05-B	24×7	Degraded
SW-H3-LEAF-07	H3/Fabric	NVIDIA Spectrum-X	—	Reachable	NOS 4.12	Premium	At risk

Device 360 — sim-node-01 MAAS + NetBox pattern

LifecycleReady

CPU2 cores

Memory4 GiB

Storage42.9 GB

Interfaces

Normalized by VM100

Interface	Purpose	MAC	Network	State
bmc	Out-of-band	02:b3:00:00:00:01	172.30.15.0/24	Up
pxe0	Provisioning	02:b3:01:00:00:01	10.10.30.0/24	Up
data0	Data	02:b3:02:00:00:01	172.30.16.0/24	Reference

Authoritative Relationships

Sitemock-dc1

Rackrack-a01

U positionU1 / 10U

Inventory sourceNetBox

Provisioning sourceMAAS

Power sourceRedfish

PSI authorityVM100 / PostgreSQL

NetBoxMAASRedfishKea

System Health & Heartbeat

PSI PULSE component state and infrastructure health projected through VM100. Proxmox-style VM metrics are shown as a reference visualization, not a direct frontend connection.

VM100 projection

FrontendHealthy

UI build / API client / auth gate

VM100Healthy

Core API / control plane

VM101Healthy

Simulator / discovery / protocol execution

PostgreSQLHealthy

Canonical operational authority

Alloy Metrics/LogsHealthy

Collector pipeline

Prometheus / MimirHealthy

Metrics query / remote write

LokiHealthy

Operational logs

AlertmanagerDegraded

1 route awaiting target evidence

DCGMHealthy

GPU metrics source

RedfishHealthy

16 simulated BMC endpoints

NetBoxReference

Inventory / rack / device context

MAASReference

Commission / deploy lifecycle

Proxmox VM Reference

VM	Purpose	CPU	RAM	Network	Status
vm-devops	NetBox / Gitea / OpenBao	2 vCPU	4 GiB	mgmt + data	Running
vm-oobsvc	Kea DHCP / OOB services	2 vCPU	2 GiB	mgmt + OOB	Running
vm-maas	MAAS / PXE provisioning	2 vCPU	4 GiB	mgmt + OOB + PXE	Running

Freshness & Dependency

Heartbeat freshness target< 60 sec

Telemetry qualityVALID

Last VM101 ingest8 sec ago

PostgreSQL authorityAvailable

Direct FE → Proxmox/MAASBlocked by design

No false green

If VM100 cannot confirm freshness/dependency state, the frontend must show Unknown/Degraded—not Healthy.

Network Fabric

Leaf-spine, rail, plane, BER, congestion, optics and blast radius.

Availability

99.97%

Healthy Links

12,274

BER Warnings

14

Congested Ports

8

Plane Resilience

A/B

Storage Alerts

2

Hall 3 Topology

Spine-01

Spine-02

Spine-03

Spine-04

Leaf-01

Leaf-02

Leaf-07

Leaf-08

Rail-A1

Rail-A2

Rail-B1

Rail-B2

Rack-07

Rack-08

Rack-09

Rack-10

L1 Fabric Checks

Leaf-07 BER validation

Inspect optic temperature, port counters, cable batch and redundant path state.

Plane A/B healthy

No current single-plane exposure.

Real Lab Reference

B300 cluster lifecycle reference used to make PSI PULSE screens and configuration more realistic. This is a reference environment, not the production architecture.

Source-backed configuration

Host-1 — Proxmox / Control Services

vm-devopsNetBox :8000 • Gitea :3000 • OpenBao :8200

vm-oobsvcKea DHCP on OOB / BMC network

vm-maasMAAS enlist / commission / deploy • OOB + PXE

NetBoxGiteaOpenBaoKeaMAAS

⇄

Host-2 — Simulated B300 Rack

sim-node-01 … sim-node-164 racks • U1/U11/U21/U31 • HGX B300 simulated nodes

sushy-tools Redfish BMCPower control / BMC API behavior used by MAAS

PXE / provisioning path10.10.30.0/24 back-to-back provisioning network

RedfishPXESimulated BMC

Network Plan

Plane	Subnet / VLAN	Use
Management	172.30.9.0/24	Host / VM management
OOB / BMC	172.30.15.0/24 • VLAN 2315	Kea, Redfish, MAAS BMC reachability
Data	172.30.16.0/24 • VLAN 2316	Service / data path
Provision / PXE	10.10.30.0/24	MAAS enlist/deploy

Zero-Touch Reference Flow

BMC lease→Kea hook→Discovery→NetBox staged→MAAS commission→Deploy→NetBox active

PSI PULSE adoption

VM101 can emulate protocol/discovery behavior; VM100 remains canonical workflow and evidence authority. The frontend receives normalized states only from VM100.

Integration Translation into PSI PULSE

External system	Reference responsibility	PSI PULSE surface	Boundary
NetBox	Inventory, site, rack, device/interface context	Assets / Topology / Device 360	Adapter → VM100
MAAS	Enlist, commission, deploy lifecycle	Onboarding / Provisioning	Adapter → VM100
Kea DHCP	Lease / discovery trigger	Discovery candidate / evidence	VM101 or integration → VM100
Redfish	BMC power / sensors / events	Device health / diagnostics	VM101 execution → VM100
Proxmox	Lab VM / infrastructure status	System Health	Integration → VM100
DCGM / Grafana	GPU telemetry / reference visualization	GPU Overview / Alerts	Telemetry → VM100 projection

Incidents

Correlated L1 incidents with owner, SLA, evidence and approvals.

Detected

→

Validated

→

Correlated

→

Investigating

→

Escalated

→

Monitoring

→

Closed

Incident Queue

P2

Repeated XID 31 on four GPUs

INC-2472 • Compute Ops • Evidence pending

03:11

P2

Leaf-07 BER trend

Network Ops • OEM case recommended

00:42

Incident Detail

Impact42 nodes at risk

Probable causeOptic / cable degradation

Confidence86%

Next actionValidate optic telemetry

ApprovalNot required for diagnostic

Runbooks

Signed diagnostics, controlled actions and restricted procedures.

Safe Diagnostics

RB-GPU-001 — DCGM Health Check

Read-only GPU health, XID, ECC, NVLink and thermal state.

Pre-approved

RB-BMC-002 — Redfish Event Bundle

Collect BMC events and sensor state.

Pre-approved

Controlled Actions

RB-COL-004 — Restart Collector

Peer and replication pre-check required.

1 approval

RB-GW-006 — Gateway Failover

Fail over hall gateway and verify replay.

2 approvals

Restricted

GPU Reset

Workload drain, change ticket and two-person approval.

Restricted

Fabric Configuration Change

Maintenance window and rollback required.

Restricted

Audit & Reports

Immutable event history, evidence and management reporting.

Audit Completeness

99.8%

Privileged Events

18

Blocked Commands

3

Prevented Risks

7

SLA Met

98.6%

Chain Verification

14:31

Recent Audit Events

AUTH

ACT-00482 approved

MFA and device binding verified

14:22

EXEC

Collector restart completed

Post-check passed

14:23

BLOCK

Unsigned GPU reset blocked

Missing change ticket and second approval

13:48

Available Reports

Daily Operations Report

Weekly Service Review

Monthly Executive Pack

OEM Performance

Pulse Agent Assurance

Settings & Configuration

Users, roles, observability, visual mapping, thresholds, approvals, Pulse Agent and security.

Users & Role Profiles

Demo credentials are visible only in this prototype. Production must use password hashes and MFA.

Username	Demo Password	Role	Default Route	MFA	Authority

Platform Integrations — Frontend via VM100 API Design-controlled / Runtime pending

Frontend hanya mengakses VM100 API/facade. Web, Console dan Git dibuka sebagai controlled launch target. Token, API key, SSH key dan privileged credential tidak disimpan di browser.

Platform	Required Surface	Current VM100 API	API Baru	Native Launch	Status	Last Test	Actions

State Model

NOT_CONFIGURED→CONFIGURED→REACHABLE→AUTHENTICATED→AUTHORIZED→VERIFIED

VM100 Candidate Contract

GET /api/v1/platform-integrations GET /api/v1/platform-integrations/{id} GET /api/v1/platform-integrations/{id}/health GET /api/v1/platform-integrations/{id}/capabilities GET /api/v1/platform-integrations/{id}/launch-targets POST /api/v1/platform-integrations/{id}/test-connection PATCH /api/v1/platform-integrations/{id}/configuration

Guardrail

No direct vendor call

Browser tidak melakukan fetch langsung ke OpenBao, Kea, MAAS/BCM, NetBox, Gitea, Grafana atau host SSH/VM.

Real Lab Reference Profile Reference-only

This profile models the B300 lab configuration. Frontend fields describe an integration profile only; browsers must not call NetBox, MAAS, Proxmox, Redfish, or telemetry backends directly.

Profile

Canonical API gateway

Site

Device model

Management subnet

OOB/BMC subnet

Data subnet

Provision/PXE subnet

Inventory & provisioning

NetBox + MAAS + Kea are modeled as server-side integrations owned by VM100/VM101 adapters.

NetBoxMAASKea

Protocol execution

Redfish/sushy-tools belongs to VM101 discovery/protocol execution. Secrets remain server-side.

RedfishVM101OpenBao

Guardrail: Save Configuration persists a VM100 integration profile. It never stores root tokens or BMC passwords in browser-local state and never turns this page into a direct infrastructure client.

Observability Configuration Wizard

1. DCGM ExporterGPU telemetry endpoint

2. Prometheus / MimirScrape and retention

3. GrafanaData sources and provisioning

4. Logs & AlertsLoki/OpenSearch, Alertmanager

5. Assets & LabelsLocation and workload context

6. Readiness TestConnectivity and quality

DCGM Exporter

Deployment

Metrics port

Metrics path

Collection interval

Metrics config

MIG pod mapping

Required groups: utilization, memory, temperature, power, energy, clocks, XID, ECC, PCIe, NVLink and MIG identifiers.

Prometheus / Mimir

Query endpoint via VM100

Scrape interval

Remote write

Hot retention

Alertmanager

Recording rules

scrape_configs: - job_name: dcgm-exporter scrape_interval: 15s metrics_path: /metrics static_configs: - targets: ["hgx-node-001:9400"] labels: site: batam hall: hall-4 cluster_id: hgx-b300-prod

Grafana

Grafana reference URL (server-side)

Organization

Data source UID

Dashboard folder

Reference dashboard

Provisioning

apiVersion: 1 datasources: - name: PSI Prometheus uid: psi-prometheus type: prometheus access: proxy url: https://prometheus.internal isDefault: true

Logs, Alerts & ITSM

Log platform

Log query via VM100

Alert route

ITSM

VM100 event ingress

Authentication

Mandatory Context Model

Group	Required labels	Purpose
Physical	site, building, hall, row, rack, u_position	Heat map and drill-down
Asset	asset_id, hostname, serial, model, OEM	Inventory and ownership
GPU/MIG	gpu, UUID, mig_profile, gpu_instance_id, compute_instance_id	GPU/MIG panels
Cluster	cluster_id, namespace, pod, container	Workload impact
Network	switch_id, port_id, peer_device, plane, rail	Topology and blast radius
Operations	severity, owner, maintenance_state, change_id	Routing and audit

Readiness Test

DCGM endpoint

HTTP 200 and DCGM_FI_* metrics expected.

Sample connected

Prometheus query

count(DCGM_FI_DEV_GPU_UTIL) by (hall)

Sample passed

Grafana data source

Health check and variable population.

Real URL required

Label completeness

Target ≥95% for site, hall, rack and asset labels.

Validation pending

Visualization Mapping

GPU gauges

Prometheus + DCGM utilization and memory metrics.

Thermal/power trends

DCGM time-series with warning and critical thresholds.

MIG allocation

DCGM + Kubernetes pod-resource labels.

Rack heat map

Asset registry + correlated health score.

Fabric topology

NetQ/UFM/streaming telemetry + CMDB.

Incident queue

Alertmanager + PostgreSQL incident workflow.

OEM compliance

Redfish/OEM API + approved firmware baseline.

Pulse Agent findings

Metrics, logs, topology, incidents and change context.

Audit timeline

PostgreSQL audit ledger + evidence object store.

Command Center Display

Refresh

Resolution

Visible halls

Critical overlay

Thresholds & Severity

GPU warning / critical

Telemetry freshness

XID recurrence

BER trend

Collector scale-out

Queue lag warning

Approval Matrix

Safe diagnosticsPolicy pre-approved

Collector restartShift Lead

Gateway failoverShift Lead + Service Owner

GPU resetService Owner + OEM/L2

Fabric/mass actionTwo-person + change ticket

Pulse Agent

Operating mode

Preventive monitoring
Evidence and confidence required
Treat logs as untrusted input
Controlled remediation

Security & Session

MFA

Session timeout

Password storage

Audit retention

Device binding
Allowlisted mobile numbers
Independent kill switch

Full-spectrum L1 operations for the AI factory.

Sign in

GPU / MIG Operations Dashboard

Aggregate

Per GPU

Per MIG Profile

Error & Reliability

L1 Operations Queue

Multi-Hall Fleet Status

Active Incident Queue

Fleet Availability

GPU Capacity

Pulse Agent Early Warning

Readiness

Onboarding & Provisioning

Fleet & Assets

Servers

GPUs

Switches

Telemetry Sources

Interfaces

System Health & Heartbeat

Network Fabric

Real Lab Reference

Host-1 — Proxmox / Control Services

Host-2 — Simulated B300 Rack

Incidents

Runbooks

RB-GPU-001 — DCGM Health Check

RB-BMC-002 — Redfish Event Bundle

RB-COL-004 — Restart Collector

RB-GW-006 — Gateway Failover

GPU Reset

Fabric Configuration Change

Audit & Reports

Settings & Configuration

State Model

VM100 Candidate Contract

Guardrail

Inventory & provisioning

Protocol execution

DCGM Exporter

Prometheus / Mimir

Grafana

Logs, Alerts & ITSM

Mandatory Context Model

Readiness Test

DCGM endpoint

Prometheus query

Grafana data source

Label completeness

GPU gauges

Thermal/power trends

MIG allocation

Rack heat map

Fabric topology

Incident queue

OEM compliance

Pulse Agent findings

Audit timeline