HiCAIN Virtual Switch Design Document¶
1. Overview¶
The HiCAIN Virtual Switch (hicain-vswitchd) is a standalone C daemon running on the host OS. It emulates a Data Center Top-of-Rack (TOR) switch inspired by Open Compute Project (OCP) networking designs. It multiplexes two distinct fabric types over the same port infrastructure:
- RoCEv2 (RDMA over Converged Ethernet v2), UDP/IP-encapsulated RDMA over a lossless Ethernet fabric
- InfiniBand, native IB frames routed by Local Identifier (LID)
Scope note. This switch is the NIC fabric only (RoCEv2 and InfiniBand). GPU-to-GPU traffic uses a separate fabric, see HiLink Switch Design. HiCCL may optionally fall back to this switch's RoCE transport when HiLink is unavailable, but the two daemons are independent.
The switch provides a lossless Ethernet environment for RoCEv2 by implementing the three pillars of Data Center Bridging (DCB):
| DCB Feature | Standard | Purpose |
|---|---|---|
| Priority Flow Control (PFC) | IEEE 802.1Qbb | Per-priority PAUSE to prevent buffer overflow |
| Enhanced Transmission Selection (ETS) | IEEE 802.1Qaz | Bandwidth allocation per Traffic Class |
| Explicit Congestion Notification (ECN) | RFC 3168 | Mark (don't drop) packets on congestion |
1.1 Port Layout, 10-Port Fixed Configuration¶
The switch has a fixed 10-port layout with three distinct port roles:
| Port(s) | Role | Protocol | Description |
|---|---|---|---|
| 0, 7 | FABRIC |
RoCEv2 + InfiniBand | Connect to VMs (RoCE-IB-vNICs) or to another TOR switch for rack aggregation |
| 8 | UPLINK |
Ethernet only | Connect to upstream network / Internet gateway |
| 9 | CONSOLE |
Observability | Dedicated OpenTelemetry telemetry export port |
#define HICAIN_NUM_FABRIC_PORTS 8 /* Ports 0–7: RoCEv2 / IB fabric */
#define HICAIN_UPLINK_PORT 8 /* Port 8: Ethernet uplink */
#define HICAIN_CONSOLE_PORT 9 /* Port 9: Observability */
#define HICAIN_TOTAL_PORTS 10
Port role constraints:
- FABRIC ports (0, 7): Support STRICT_ETH, STRICT_IB, or AUTO mode. Full DCB (PFC/ETS/ECN) support. Can be used for VM connections OR inter-switch (TOR-to-TOR) links.
- UPLINK port (8): Fixed STRICT_ETH mode. No IB. Standard L2/L3 forwarding to external networks. No PFC (best-effort upstream).
- CONSOLE port (9): Not a data-plane port. Emits OpenTelemetry telemetry (metrics, traces, logs) over OTLP/gRPC or OTLP/HTTP to an external collector.
1.2 TOR-to-TOR Aggregation (OCP Rack Interconnect)¶
Two HiCAIN switches can be connected together to form a two-rack aggregated fabric, mirroring how OCP TOR switches interconnect:
flowchart LR
subgraph RA["Rack A"]
SA["TOR Switch A"]
A0["Port 0<br/>VM A0"]
A1["Port 1<br/>VM A1"]
A2["Port 2<br/>VM A2"]
A3["Port 3<br/>VM A3"]
A4["Port 4<br/>VM A4"]
A5["Port 5<br/>VM A5"]
A6["Port 6<br/>ISL"]
A7["Port 7<br/>ISL"]
A8["Port 8<br/>Uplink - Internet"]
A9["Port 9<br/>Console - OTel"]
SA --- A0 & A1 & A2 & A3 & A4 & A5 & A6 & A7 & A8 & A9
end
subgraph RB["Rack B"]
SB["TOR Switch B"]
B0["Port 0<br/>VM B0"]
B1["Port 1<br/>VM B1"]
B2["Port 2<br/>VM B2"]
B3["Port 3<br/>VM B3"]
B4["Port 4<br/>VM B4"]
B5["Port 5<br/>VM B5"]
B6["Port 6<br/>ISL"]
B7["Port 7<br/>ISL"]
B8["Port 8<br/>Uplink - Internet"]
B9["Port 9<br/>Console - OTel"]
SB --- B0 & B1 & B2 & B3 & B4 & B5 & B6 & B7 & B8 & B9
end
A6 === B6
A7 === B7
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
class SA,SB blue
class A0,A1,A2,A3,A4,A5,B0,B1,B2,B3,B4,B5 green
class A6,A7,B6,B7 amber
In this topology: - Ports 0, 5 on each switch connect to VMs (6 VMs per rack, 12 total) - Ports 6, 7 are inter-switch links (ISLs) connecting the two TOR switches (2 links for redundancy/bandwidth) - Frames destined for a VM on the remote rack are forwarded across the ISL - The switch treats ISL ports identically to VM ports, no special ISL logic is needed. The FDB/LFT naturally learns that remote MACs/LIDs are reachable via the ISL port
ISL and IB routing: For InfiniBand, the LFT on each switch must have entries pointing remote LIDs to the ISL port(s). For Ethernet, MAC learning across the ISL works automatically.
ISL link aggregation (optional v2): Ports 6, 7 could be bonded into a LAG for increased bandwidth and failover. For v1, they operate as independent links with static FDB/LFT entries.
2. Architecture Overview¶
flowchart TB
subgraph HOST["hicain-vswitchd host"]
CP["Control Plane<br/>mgmt.sock<br/>JSON management"]
subgraph DP["Data Plane"]
P0["Port 0"]
P1["Port 1"]
P2["Port 2"]
P3["Port 3"]
P4["Port 4"]
P5["Port 5"]
P6["Port 6"]
P7["Port 7"]
CLASS["Frame Classifier<br/>Ethernet or IB LRH"]
IB["IB Pipeline<br/>LID routing"]
ETH["RoCEv2/Eth Pipeline<br/>MAC/VLAN + DCB"]
P8["Port 8<br/>Ethernet uplink"]
P9["Port 9<br/>Observability"]
P0 & P1 & P2 & P3 & P4 & P5 & P6 & P7 --> CLASS
CLASS --> IB
CLASS --> ETH
P8 --> ETH
P9 --> OTEL["OTel export"]
end
end
P8 --> UP["Upstream network<br/>Internet or gateway"]
OTEL --> COL["OpenTelemetry Collector<br/>Jaeger / Prometheus"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
classDef indigo fill:#e0e7ff,stroke:#3730a3,color:#1e1b4b
class CP blue
class P0,P1,P2,P3,P4,P5,P6,P7,P8,P9 green
class CLASS,IB,ETH amber
class COL,OTEL indigo
3. Wire Protocol, Socket Framing¶
3.1 Socket Type¶
Each port uses a SOCK_SEQPACKET UNIX domain socket.
Rationale:
- Preserves message boundaries (each send() = one frame), unlike SOCK_STREAM
- No need for a length-prefix framing layer, the kernel guarantees atomic message delivery
- Behaves like a datagram but is connection-oriented (we know when an endpoint disconnects)
3.2 Frame Envelope¶
Every message on the wire is a raw L2 frame, either an Ethernet frame or an IB LRH frame. There is no additional wrapper or header added by the transport. The switch classifies based on the first bytes of the raw payload.
flowchart TB
MSG["SOCK_SEQPACKET Message"] --> FRAME["Bytes 0..N<br/>Raw L2 frame"]
FRAME --> KIND["Ethernet frame<br/>or IB LRH frame"]
MSG --> MAX["Max frame size<br/>9216 bytes jumbo"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
class MSG blue
class FRAME,KIND,MAX green
3.3 Socket Path Convention¶
/var/run/hicain/port_0.sock .. port_7.sock — Fabric ports (RoCEv2/IB)
/var/run/hicain/port_8.sock — Uplink port (Ethernet only)
/var/run/hicain/mgmt.sock — Control plane management socket
Port 9 (Console) does not use a UNIX data-plane socket. It exports telemetry via OTLP (see §15).
4. Frame Classification (Multiplexing)¶
When a frame arrives on any port, the switch must determine if it is Ethernet or InfiniBand. The classification is based on the port mode and the frame's first bytes.
4.1 Port Modes¶
| Mode | Behavior |
|---|---|
STRICT_ETH |
All frames treated as Ethernet. Non-Ethernet frames are dropped and counted. |
STRICT_IB |
All frames treated as InfiniBand LRH. Non-IB frames are dropped and counted. |
AUTO |
Inspect first bytes to classify. See §4.2. |
4.2 AUTO Classification Heuristic¶
InfiniBand LRH (Local Routing Header) is 8 bytes:
Bits [0:3] = Version (0x0 for IB)
Bits [4:7] = VL (Virtual Lane)
Bits [8:11] = LVer (Link Version, 0x0)
Bits [12:13] = SL (Service Level)
Bits [14:15] = Reserved (0b00)
Bits [16:27] = DLID (Destination LID)
Bits [28:31] = Reserved (0b0000)
Bits [32:36] = Packet Length (in 4-byte words)
...
Bits [48:63] = SLID (Source LID)
Ethernet frames start with a 6-byte Destination MAC address.
Classification algorithm:
flowchart TD
START["Classify incoming frame"] --> MODE{"Port mode?"}
MODE -->|STRICT_ETH| ETH["Ethernet pipeline"]
MODE -->|STRICT_IB| IB["IB pipeline"]
MODE -->|AUTO| BYTE["Read frame byte 0"]
BYTE --> VER["Extract IB version"]
VER --> CHECK{"Version is 0x0<br/>and length >= 8?"}
CHECK -->|Yes| LRH["Validate LRH<br/>reserved fields"]
LRH --> IB
CHECK -->|No| ETH
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
class START,BYTE,VER blue
class ETH,IB green
class MODE,CHECK,LRH amber
Note: In a real switch this heuristic would be more nuanced. For our educational platform, the clear separation of port modes plus the version-field check is sufficient and keeps the code readable.
5. InfiniBand Pipeline¶
5.1 LRH-Based Routing¶
The IB pipeline extracts the Destination LID (DLID) from the LRH (bits 16, 27) and looks it up in the LID Forwarding Table (LFT).
struct ib_lft_entry {
uint16_t dlid; /* Destination LID */
uint8_t out_port; /* Egress port number */
};
The LFT is a simple array indexed by LID (LIDs are 16-bit, but the educational switch can limit to a configurable max, e.g., 256 entries).
5.2 IB Forwarding Logic¶
flowchart TD
PARSE["Parse LRH<br/>extract DLID and SLID"] --> LOOKUP["Lookup DLID in LFT"]
LOOKUP -->|HIT| FWD["Forward frame<br/>to out_port"]
LOOKUP -->|MISS| DROP["Drop frame<br/>increment counter"]
PARSE --> LEARN["Optional learn SLID<br/>to ingress port"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef red fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
class PARSE,LOOKUP,LEARN blue
class FWD green
class DROP red
5.3 IB Virtual Lanes¶
The LRH contains a 4-bit Virtual Lane (VL) field. The switch respects VL15 as the management VL (always forwarded, never blocked by flow control). Other VLs map to internal queues for future QoS extensions.
6. RoCEv2 / Ethernet Pipeline¶
6.1 Frame Format (RoCEv2)¶
| Field | Size / value | Notes |
|---|---|---|
| Dst MAC | 6 bytes | Destination Ethernet MAC |
| Src MAC | 6 bytes | Source Ethernet MAC |
| 802.1Q Tag | 4 bytes | Optional VLAN tag |
| EtherType | 0x0800 |
IPv4 |
| IP Header | 20 bytes | Carries ECN bits |
| UDP | 8 bytes | Destination port 4791 |
| IB | Variable | BTH and payload |
- 802.1Q VLAN Tag (optional, 4 bytes): Contains the 3-bit PCP (Priority Code Point) field used by PFC and ETS.
- EtherType
0x0800= IPv4 (RoCEv2 rides over IP). - UDP destination port
4791= IANA-assigned RoCEv2 port. - IP Header carries the ECN bits (bits 6, 7 of the TOS/DSCP byte).
6.2 MAC Forwarding Table¶
struct mac_fdb_entry {
uint8_t mac[6]; /* MAC address */
uint16_t vlan_id; /* VLAN ID (0 = untagged) */
uint8_t out_port; /* Egress port number */
uint32_t age; /* Aging timer (seconds since last seen) */
};
Forwarding logic:
flowchart TD
PARSE["Parse Ethernet header<br/>Dst MAC and VLAN"] --> LOOKUP["Lookup MAC + VLAN<br/>in FDB"]
LOOKUP -->|HIT| FWD["Forward to out_port"]
LOOKUP -->|MISS| FLOOD["Flood in same VLAN<br/>except ingress"]
PARSE --> LEARN["Learn Src MAC<br/>to ingress port"]
PARSE --> VLAN{"VLAN tagged?"}
VLAN -->|Yes| PCP["Extract PCP<br/>for QoS"]
VLAN -->|No| SKIP["Skip QoS priority<br/>classification"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
class PARSE,LOOKUP,LEARN blue
class FWD,PCP green
class FLOOD,VLAN,SKIP amber
6.3 Broadcast & Multicast¶
- Broadcast (
FF:FF:FF:FF:FF:FF): Flood to all ports in the same VLAN (except ingress). - Multicast: Flood to all ports (no IGMP snooping in v1).
7. Data Center Bridging (DCB), Lossless Ethernet¶
7.1 Priority Flow Control (PFC), IEEE 802.1Qbb¶
PFC enables per-priority PAUSE, allowing the switch to halt traffic on a specific priority class without affecting other priorities. This is the foundation of lossless Ethernet for RoCEv2.
7.1.1 Priority Model¶
The 802.1Q PCP field (3 bits) defines 8 priorities (0, 7). Typically, RoCEv2 traffic is assigned to priority 3 or priority 4 (configurable). PFC is enabled only on the priorities carrying RDMA traffic.
7.1.2 Per-Priority Queues¶
Each port maintains 8 ingress queues (one per priority):
#define HICAIN_NUM_PRIORITIES 8
struct pfc_queue {
uint8_t priority; /* 0–7 */
uint32_t buffer_size; /* Max bytes in this queue */
uint32_t occupancy; /* Current bytes in queue */
uint32_t xoff_threshold; /* Send PAUSE when occupancy >= this */
uint32_t xon_threshold; /* Send un-PAUSE when occupancy <= this */
bool paused_by_peer; /* We received PAUSE from peer for this priority */
bool pause_sent; /* We sent PAUSE to peer for this priority */
};
struct port_queues {
struct pfc_queue pfc[HICAIN_NUM_PRIORITIES];
};
7.1.3 PFC PAUSE Frame Format¶
PFC uses a standard Ethernet MAC Control frame:
Dst MAC: 01:80:C2:00:00:01 (reserved multicast)
Src MAC: <port MAC>
EtherType: 0x8808 (MAC Control)
Opcode: 0x0101 (PFC)
Priority Enable Vector: 8-bit bitmap (one bit per priority)
Time[0..7]: 16-bit quanta per priority (pause duration)
7.1.4 PFC State Machine (per port, per priority)¶
stateDiagram-v2
[*] --> FLOWING
FLOWING --> PAUSED: occupancy >= xoff_threshold
PAUSED --> FLOWING: occupancy <= xon_threshold
note right of PAUSED
Send PFC PAUSE upstream
time[priority] = 0xFFFF
end note
note left of FLOWING
Send PFC PAUSE upstream
time[priority] = 0x0000
end note
When the switch receives a PFC PAUSE from an endpoint:
- Set paused_by_peer = true for the indicated priorities
- Stop transmitting frames of those priorities to that port
- Resume when time expires or a PAUSE with time=0 is received
7.2 Enhanced Transmission Selection (ETS), IEEE 802.1Qaz¶
ETS controls bandwidth allocation across Traffic Classes (TCs). It determines how the switch schedules frames from different priority queues for transmission on each port.
7.2.1 Traffic Class Mapping¶
Each of the 8 priorities is mapped to one of 8 Traffic Classes (TC0, TC7):
struct ets_config {
uint8_t prio_to_tc[HICAIN_NUM_PRIORITIES]; /* priority → TC mapping */
uint8_t tc_bandwidth[HICAIN_NUM_PRIORITIES]; /* % bandwidth per TC (must sum to 100) */
enum {
ETS_ALGO_STRICT, /* Strict priority (always served first) */
ETS_ALGO_ETS /* Weighted bandwidth sharing */
} tc_tsa[HICAIN_NUM_PRIORITIES]; /* Transmission Selection Algorithm per TC */
};
7.2.2 Scheduling Algorithm¶
The egress scheduler on each port:
- Strict-priority TCs are served first (highest TC number = highest priority)
- ETS TCs share remaining bandwidth according to
tc_bandwidth[]weights using Weighted Round Robin (WRR)
Example configuration for RoCEv2:
Priority 3 → TC3 (RDMA traffic) — ETS, 50% bandwidth
Priority 0 → TC0 (Best-effort) — ETS, 50% bandwidth
Priority 7 → TC7 (Network control) — Strict priority
7.3 Explicit Congestion Notification (ECN), RFC 3168¶
ECN allows the switch to signal congestion without dropping packets. This works in concert with PFC to create a fully lossless, congestion-aware fabric.
7.3.1 ECN Bits in IP Header¶
The two ECN bits in the IP TOS/DSCP field:
| ECN Bits | Meaning |
|---|---|
00 |
Not ECN-Capable Transport |
01 |
ECN-Capable Transport (ECT(1)) |
10 |
ECN-Capable Transport (ECT(0)) |
11 |
Congestion Experienced (CE) |
7.3.2 ECN Marking Logic¶
When a RoCEv2 packet (identified by UDP dst port 4791) passes through the switch:
struct ecn_config {
bool enabled; /* ECN marking enabled on this port */
uint32_t marking_threshold; /* Queue depth (bytes) at which to start marking */
uint32_t marking_probability; /* 0–100 (% chance of marking when above threshold) */
};
flowchart TD
START["RoCEv2 packet<br/>UDP dst 4791"] --> ECN{"ECN enabled?"}
ECN -->|No| FWD["Forward as-is"]
ECN -->|Yes| CAP{"Packet ECN-capable?"}
CAP -->|No| FWD
CAP -->|Yes| THRESH{"Queue >= threshold?"}
THRESH -->|No| FWD
THRESH -->|Yes| PROB{"Marking probability hit?"}
PROB -->|No| FWD
PROB -->|Yes| MARK["Set ECN bits to CE<br/>receiver sends CNP"]
MARK --> FWD2["Forward marked packet"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
class START blue
class FWD,FWD2 green
class ECN,CAP,THRESH,PROB,MARK amber
7.3.3 Interaction with PFC¶
ECN and PFC work together as a two-tier congestion management system:
flowchart LR
ZERO["0<br/>Empty queue"] -->|Queue depth increasing| ECN["ECN threshold<br/>Start CE marking"]
ECN --> PFC["PFC XOFF threshold<br/>Send PAUSE upstream"]
PFC --> FULL["Buffer full<br/>Should never reach"]
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
classDef red fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d
class ZERO green
class ECN,PFC amber
class FULL red
- ECN acts as an early warning, it tells the sender to slow down before buffers fill up.
- PFC acts as a last resort, if the sender doesn't slow down fast enough, PFC pauses the link to prevent any drops.
8. Per-Port Data Structures¶
enum port_role {
PORT_ROLE_FABRIC, /* Ports 0–7: RoCEv2 + InfiniBand */
PORT_ROLE_UPLINK, /* Port 8: Ethernet-only uplink */
PORT_ROLE_CONSOLE /* Port 9: Observability (no data plane) */
};
enum port_mode {
PORT_MODE_STRICT_ETH,
PORT_MODE_STRICT_IB,
PORT_MODE_AUTO
};
enum port_link_state {
LINK_DOWN,
LINK_UP
};
struct switch_port {
/* Identity */
uint8_t port_id;
enum port_role role; /* FABRIC, UPLINK, or CONSOLE */
char sock_path[256]; /* UNIX socket path */
int listen_fd; /* Listener socket fd */
int conn_fd; /* Connected endpoint fd (-1 if no endpoint) */
/* Configuration (FABRIC and UPLINK only) */
enum port_mode mode; /* UPLINK is always STRICT_ETH */
enum port_link_state link_state;
uint8_t mac[6]; /* Port's own MAC address */
/* DCB Configuration (FABRIC ports only) */
bool pfc_enabled[HICAIN_NUM_PRIORITIES]; /* Per-priority PFC enable */
struct ets_config ets;
struct ecn_config ecn;
/* Queuing */
struct port_queues queues;
/* Telemetry Counters */
struct port_stats {
uint64_t rx_frames;
uint64_t tx_frames;
uint64_t rx_bytes;
uint64_t tx_bytes;
uint64_t rx_drops; /* Frames dropped (unknown dest, full buffer, etc.) */
uint64_t tx_drops;
uint64_t pfc_pause_sent; /* Number of PFC PAUSE frames sent */
uint64_t pfc_pause_received; /* Number of PFC PAUSE frames received */
uint64_t ecn_marked; /* Packets marked with CE */
uint64_t classify_errors; /* Frames that failed classification */
} stats;
};
9. Event Loop Architecture¶
9.1 Single-Threaded epoll¶
The switch uses a single-threaded epoll event loop. This is chosen for:
- Simplicity, no locking, no race conditions, ideal for educational code
- Sufficient performance, the switch handles emulated traffic at software speeds, not line-rate hardware
- Debuggability, deterministic execution, easy to trace with gdb
struct hicain_switch {
int epoll_fd;
int mgmt_fd; /* Management socket fd */
struct switch_port ports[HICAIN_TOTAL_PORTS];
/* Forwarding Tables */
struct mac_fdb fdb; /* MAC forwarding database */
struct ib_lft lft; /* IB LID forwarding table */
/* OpenTelemetry */
struct otel_exporter *otel; /* OTLP exporter (Port 9 console) */
/* Global Config */
char run_dir[256]; /* Runtime directory for sockets */
bool running; /* Main loop control */
};
9.2 Event Loop Pseudocode¶
1. Create epoll instance
2. Create 9 UNIX SOCK_SEQPACKET listener sockets (ports 0–8, skip port 9 console)
3. Create management listener socket
4. Initialize OpenTelemetry exporter (port 9 → OTLP endpoint)
5. Start periodic telemetry export timer (e.g., every 10s)
6. Add all listeners + timer fd to epoll
7. while (running):
events = epoll_wait(epoll_fd, ...)
for each event:
if event.fd == mgmt_listener:
accept new management connection
elif event.fd == mgmt_client:
handle_mgmt_command(fd) → JSON parse & dispatch
elif event.fd == port_listener:
accept new endpoint connection → "link up"
elif event.fd == port_data:
frame = recv(fd)
if port.role == UPLINK:
forward via Ethernet pipeline only (no IB)
else:
classify_and_forward(port, frame)
elif event.fd == otel_timer:
export_telemetry_otlp(otel) → push metrics to collector
9.3 Connection Lifecycle (Port)¶
stateDiagram-v2
[*] --> DISCONNECTED
DISCONNECTED --> LINK_UP: endpoint connects
LINK_UP --> DATA_EXCHANGE: data exchange
DATA_EXCHANGE --> LINK_DOWN: endpoint disconnects
LINK_DOWN --> [*]: fd closed or EOF
When an endpoint disconnects:
- Set port link_state = LINK_DOWN
- Flush all FDB entries learned on that port
- Reset PFC state for that port
10. Control Plane, Management API¶
Full reference: See HiCAIN_Control_Plane.md for the complete control plane reference including all CLI commands, Web UI REST endpoints, configuration parameters, and operational workflows.
This section provides a summary of the management API. The control plane document is the authoritative reference.
10.1 Protocol¶
- Transport: UNIX domain socket (
SOCK_STREAM), path:/var/run/hicain/mgmt.sock - Encoding: JSON, newline-delimited (one JSON object per line)
- Pattern: Request-response (client sends command, switch sends reply)
10.2 Command Schema¶
10.2.1 Query Port Status¶
// Request
{"cmd": "port_status", "port": 0}
// Response
{
"status": "ok",
"port": 0,
"link_state": "UP",
"mode": "STRICT_ETH",
"mac": "02:00:00:00:00:01",
"stats": {
"rx_frames": 15432,
"tx_frames": 15100,
"rx_bytes": 4523100,
"tx_bytes": 4418000,
"rx_drops": 3,
"pfc_pause_sent": 12,
"pfc_pause_received": 8,
"ecn_marked": 42
}
}
10.2.2 Set Port Mode¶
{"cmd": "set_port_mode", "port": 0, "mode": "STRICT_ETH"}
{"cmd": "set_port_mode", "port": 1, "mode": "STRICT_IB"}
{"cmd": "set_port_mode", "port": 2, "mode": "AUTO"}
10.2.3 Set Link State¶
{"cmd": "set_link_state", "port": 0, "state": "UP"}
{"cmd": "set_link_state", "port": 0, "state": "DOWN"}
10.2.4 Configure PFC¶
{"cmd": "set_pfc", "port": 0, "priorities": [3, 4], "enabled": true}
{"cmd": "set_pfc", "port": 0, "priorities": [0, 1, 2, 5, 6, 7], "enabled": false}
10.2.5 Configure ETS¶
{
"cmd": "set_ets",
"port": 0,
"prio_to_tc": [0, 0, 0, 3, 3, 0, 0, 7],
"tc_bandwidth": [50, 0, 0, 50, 0, 0, 0, 0],
"tc_tsa": ["ets", "strict", "strict", "ets", "strict", "strict", "strict", "strict"]
}
10.2.6 Configure ECN¶
{"cmd": "set_ecn", "port": 0, "enabled": true, "marking_threshold": 32768, "marking_probability": 10}
10.2.7 Manage Forwarding Tables¶
// Add static MAC FDB entry
{"cmd": "fdb_add", "mac": "02:00:00:00:00:01", "vlan": 100, "port": 0}
// Delete MAC FDB entry
{"cmd": "fdb_del", "mac": "02:00:00:00:00:01", "vlan": 100}
// Dump MAC FDB
{"cmd": "fdb_dump"}
// Add IB LFT entry
{"cmd": "lft_add", "dlid": 1, "port": 0}
// Dump IB LFT
{"cmd": "lft_dump"}
10.2.8 Dump All Telemetry¶
{"cmd": "telemetry_dump"}
// Response: array of all port stats (same data exported via OpenTelemetry on port 9)
11. Management Interfaces, Web UI & CLI¶
Both the Web UI and CLI are clients of the management socket API defined in §10. They translate user actions into JSON commands sent over /var/run/hicain/mgmt.sock, keeping the C switch daemon free of any UI concerns.
flowchart TB
CLI["hicain-cli<br/>Python"] -->|JSON over UNIX<br/>domain socket| SW["hicain-vswitchd<br/>mgmt.sock control plane"]
WEB["hicain-webui<br/>Python + React/HTML"] -->|JSON over UNIX<br/>domain socket| SW
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
class CLI,WEB blue
class SW green
11.1 CLI, hicain-cli¶
A Python command-line tool for managing the switch from the terminal. It connects to the management UNIX socket via asyncio, sends JSON commands, and pretty-prints the responses.
11.1.1 Python Toolchain¶
| Tool | Purpose |
|---|---|
| Typer | CLI framework (type-hint driven, auto-generates --help) |
| asyncio | Async I/O for non-blocking socket communication |
| Pydantic | Data validation & serialization for API request/response models |
| Rich | Terminal formatting (tables, syntax highlighting, progress bars) |
| Ruff | Linting & formatting (replaces flake8, black, isort) |
| Poetry | Dependency management & packaging |
| pytest + pytest-asyncio | Testing (async-aware) |
| mypy | Static type checking |
All Python code uses strict static typing, no Any types, all function signatures fully annotated.
11.1.2 Command Structure¶
The CLI uses a <resource> <action> [options] pattern inspired by ip and mlxconfig:
# Port management
hicain-cli port status # Show all ports
hicain-cli port status 0 # Show port 0 details
hicain-cli port set-mode 0 STRICT_ETH # Set port 0 to Ethernet-only
hicain-cli port set-mode 2 AUTO # Set port 2 to auto-detect
hicain-cli port link-up 0 # Administratively bring port 0 up
hicain-cli port link-down 0 # Administratively bring port 0 down
# DCB — Priority Flow Control
hicain-cli pfc enable 0 --priorities 3,4 # Enable PFC on port 0, priorities 3 and 4
hicain-cli pfc disable 0 --priorities 0,1,2,5,6,7
hicain-cli pfc status 0 # Show PFC state per priority
# DCB — Enhanced Transmission Selection
hicain-cli ets set 0 \
--prio-tc 0:0,1:0,2:0,3:3,4:3,5:0,6:0,7:7 \
--tc-bw 0:50,3:50 \
--tc-tsa 0:ets,3:ets,7:strict
hicain-cli ets status 0 # Show ETS config on port 0
# DCB — ECN
hicain-cli ecn set 0 --enable --threshold 32768 --probability 10
hicain-cli ecn status 0
# Forwarding tables
hicain-cli fdb show # Dump MAC forwarding database
hicain-cli fdb add --mac 02:00:00:00:00:01 --vlan 100 --port 0
hicain-cli fdb del --mac 02:00:00:00:00:01 --vlan 100
hicain-cli lft show # Dump IB LID forwarding table
hicain-cli lft add --dlid 1 --port 0
# Telemetry
hicain-cli telemetry show # Dump all port counters
hicain-cli telemetry clear # Reset all counters
# Global
hicain-cli --socket /path/to/mgmt.sock ... # Override default socket path
11.1.3 Output Formatting¶
The CLI outputs human-readable tables by default and supports --json for machine-readable output:
$ hicain-cli port status
PORT ROLE MODE LINK RX-FRAMES TX-FRAMES RX-DROPS PFC-TX ECN-MARK
──── ─────── ────────── ───── ───────── ───────── ──────── ────── ────────
0 FABRIC AUTO UP 15432 15100 3 12 42
1 FABRIC AUTO UP 8210 8190 0 0 0
2 FABRIC STRICT_IB UP 4500 4500 0 0 0
3 FABRIC AUTO DOWN 0 0 0 0 0
4 FABRIC AUTO DOWN 0 0 0 0 0
5 FABRIC AUTO DOWN 0 0 0 0 0
6 FABRIC AUTO UP 2100 2050 1 2 0
7 FABRIC AUTO UP 2100 2040 0 3 0
8 UPLINK STRICT_ETH UP 50000 49800 10 — —
9 CONSOLE — — — — — — —
$ hicain-cli port status 0 --json
{"status": "ok", "port": 0, "role": "FABRIC", "link_state": "UP", ...}
11.2 Web UI, hicain-webui¶
A browser-based dashboard for visual switch management. It provides real-time port status, DCB configuration, forwarding table inspection, and live telemetry visualization.
11.2.1 Architecture¶
The Web UI is a Python backend (FastAPI) + Next.js/React frontend (TypeScript):
flowchart TB
subgraph BROWSER["Browser"]
FE["Frontend<br/>Next.js + React + TS"]
DASH["Dashboard + DCB panels<br/>FDB/LFT + telemetry"]
TOPO["Topology diagram<br/>React Flow"]
FE --- DASH
FE --- TOPO
end
FE -->|REST API + WebSocket| BE["hicain-webui backend<br/>Python FastAPI"]
BE --> API["Translate REST to<br/>JSON mgmt commands"]
BE --> WS["WebSocket proxy<br/>real-time telemetry"]
BE --> MODELS["Pydantic models<br/>request/response"]
BE -->|UNIX socket| SW["hicain-vswitchd<br/>mgmt.sock"]
BE --> URL["Default URL<br/>localhost:8080"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
class FE,DASH,TOPO blue
class BE,API,WS,MODELS amber
class SW,URL green
11.2.2 Technology Stack¶
Backend (Python):
| Tool | Purpose |
|---|---|
| FastAPI | Async REST API + WebSocket proxy |
| Uvicorn | ASGI server |
| Pydantic | Request/response validation (shared models with CLI) |
| asyncio | Non-blocking mgmt.sock communication |
| Poetry | Dependency management (shared pyproject.toml with CLI) |
| Ruff | Linting & formatting |
| pytest + httpx | API integration testing |
Frontend (TypeScript):
| Tool | Purpose |
|---|---|
| Next.js 14+ | React framework (App Router, SSR/SSG) |
| React 18+ | UI component library |
| TypeScript | Static typing |
| Recharts | Telemetry charts (time-series, gauges) |
| React Flow | Topology diagram (TOR-to-TOR visualization) |
| Tailwind CSS | Utility-first styling |
| SWR / React Query | Data fetching & cache (REST polling + WebSocket) |
| shadcn/ui | Accessible UI components (tables, forms, dialogs) |
- The Web UI is a separate process from the switch daemon, clean separation of concerns
- The FastAPI backend and Next.js frontend can be developed independently
- Next.js can be deployed as a static export (
next export) or server-rendered - Students can study the REST→UNIX-socket proxy pattern
11.2.3 REST API Endpoints¶
The Web UI backend exposes REST endpoints that map 1:1 to the management socket JSON commands:
| Method | Endpoint | Maps to mgmt cmd | Description |
|---|---|---|---|
GET |
/api/ports |
port_status (all) |
List all 10 ports with status |
GET |
/api/ports/{id} |
port_status |
Single port details |
PUT |
/api/ports/{id}/mode |
set_port_mode |
Set port mode |
PUT |
/api/ports/{id}/link |
set_link_state |
Set link state |
GET |
/api/ports/{id}/pfc |
(query) | Get PFC status for port |
PUT |
/api/ports/{id}/pfc |
set_pfc |
Configure PFC |
GET |
/api/ports/{id}/ets |
(query) | Get ETS config for port |
PUT |
/api/ports/{id}/ets |
set_ets |
Configure ETS |
GET |
/api/ports/{id}/ecn |
(query) | Get ECN config for port |
PUT |
/api/ports/{id}/ecn |
set_ecn |
Configure ECN |
GET |
/api/fdb |
fdb_dump |
Dump MAC FDB |
POST |
/api/fdb |
fdb_add |
Add FDB entry |
DELETE |
/api/fdb |
fdb_del |
Delete FDB entry |
GET |
/api/lft |
lft_dump |
Dump IB LFT |
POST |
/api/lft |
lft_add |
Add LFT entry |
GET |
/api/telemetry |
telemetry_dump |
All port counters |
WebSocket |
/ws/telemetry |
telemetry_dump (polled) |
Real-time telemetry stream |
11.2.4 Dashboard Pages¶
| Page | Description |
|---|---|
| Overview | Visual 10-port switch front panel. Each port shows link state (green/red LED), role badge, mode, and live RX/TX sparklines. |
| Port Detail | Single-port deep dive: DCB config (PFC/ETS/ECN), queue occupancy gauges, per-priority counters. |
| Forwarding Tables | Searchable/sortable tables for MAC FDB and IB LFT. Add/delete entries inline. |
| Telemetry | Real-time charts (via WebSocket): frame rates, byte throughput, PFC PAUSE events, ECN marks. Time-series with configurable window. |
| Topology | Shows TOR-to-TOR interconnect (when ISL ports are active). Visualizes which ports connect to VMs vs. ISLs. |
11.2.5 Startup¶
# Start the Web UI (connects to switch management socket)
hicain-webui --socket /var/run/hicain/mgmt.sock --listen 0.0.0.0:8080
# Or with defaults
hicain-webui # → http://localhost:8080, socket at /var/run/hicain/mgmt.sock
12. Threading & Concurrency¶
Decision: Single-threaded.
All I/O (data plane + control plane) is multiplexed on a single epoll event loop. This avoids:
- Mutexes on forwarding tables
- Race conditions in PFC state machines
- Complexity that detracts from the educational goal
Queue processing (PFC thresholds, ETS scheduling, ECN marking) is performed inline during the classify_and_forward() path. Since this is a software emulation at modest traffic rates, this is acceptable.
If performance becomes a concern in future versions, the data plane could be separated to a dedicated thread with lock-free queues.
13. Directory Structure¶
vdc/
├── AGENT_CONTEXT.md
├── mkdocs.yml ← MkDocs configuration
├── pyproject.toml ← Poetry: shared Python workspace (CLI + WebUI backend)
├── ruff.toml ← Ruff linter/formatter config
├── docs/
│ ├── index.md ← Documentation home page
│ ├── HiCAIN_Virtual_Switch_Design.md ← Switch architecture & design
│ └── HiCAIN_Control_Plane.md ← Control plane reference (CLI, WebUI, API)
├── switch/
│ ├── Makefile
│ ├── src/
│ │ ├── main.c ← Entry point, argument parsing, daemonize
│ │ ├── switch.c / switch.h ← Core switch struct, init/teardown
│ │ ├── epoll_loop.c / epoll_loop.h ← Event loop, connection management
│ │ ├── classifier.c / classifier.h ← Frame classification (ETH vs IB)
│ │ ├── eth_pipeline.c / eth_pipeline.h ← Ethernet/RoCEv2 forwarding, FDB
│ │ ├── ib_pipeline.c / ib_pipeline.h ← InfiniBand LRH forwarding, LFT
│ │ ├── pfc.c / pfc.h ← Priority Flow Control logic
│ │ ├── ets.c / ets.h ← Enhanced Transmission Selection
│ │ ├── ecn.c / ecn.h ← ECN marking logic
│ │ ├── mgmt.c / mgmt.h ← Management socket, JSON command handler
│ │ ├── otel.c / otel.h ← OpenTelemetry OTLP exporter (port 9)
│ │ └── protocol.h ← Wire format structs (Ethernet, LRH, PFC frame)
│ └── include/
│ └── hicain_common.h ← Shared constants, macros, port role definitions
├── cli/
│ ├── hicain_cli/
│ │ ├── __init__.py
│ │ ├── main.py ← Typer app entry point
│ │ ├── commands/
│ │ │ ├── __init__.py
│ │ │ ├── port.py ← port status/set-mode/link-up/link-down
│ │ │ ├── pfc.py ← pfc enable/disable/status
│ │ │ ├── ets.py ← ets set/status
│ │ │ ├── ecn.py ← ecn set/status
│ │ │ ├── fdb.py ← fdb show/add/del
│ │ │ ├── lft.py ← lft show/add/del
│ │ │ └── telemetry.py ← telemetry show/clear
│ │ ├── client.py ← Async management socket client
│ │ └── models.py ← Pydantic models (shared with webui)
│ └── tests/
│ └── test_cli.py ← CLI command tests
├── webui/
│ ├── backend/
│ │ ├── hicain_webui/
│ │ │ ├── __init__.py
│ │ │ ├── app.py ← FastAPI application
│ │ │ ├── routes/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── ports.py ← /api/ports endpoints
│ │ │ │ ├── dcb.py ← /api/ports/{id}/pfc,ets,ecn
│ │ │ │ ├── tables.py ← /api/fdb, /api/lft endpoints
│ │ │ │ └── telemetry.py ← /api/telemetry + /ws/telemetry
│ │ │ ├── client.py ← Async mgmt socket client
│ │ │ └── models.py ← Pydantic models (shared with CLI)
│ │ └── tests/
│ │ └── test_webui_api.py ← REST API integration tests
│ └── frontend/
│ ├── package.json ← Node.js dependencies
│ ├── tsconfig.json ← TypeScript configuration
│ ├── tailwind.config.ts ← Tailwind CSS config
│ ├── next.config.js ← Next.js configuration
│ ├── public/ ← Static assets
│ └── src/
│ ├── app/
│ │ ├── layout.tsx ← Root layout (navigation, theme)
│ │ ├── page.tsx ← Overview dashboard (10-port panel)
│ │ ├── port/[id]/page.tsx ← Port detail page
│ │ ├── tables/page.tsx ← FDB / LFT table viewer
│ │ ├── telemetry/page.tsx ← Real-time telemetry charts
│ │ └── topology/page.tsx ← TOR-to-TOR topology diagram
│ ├── components/
│ │ ├── port-panel.tsx ← Port status card with LED indicator
│ │ ├── pfc-config.tsx ← PFC configuration form
│ │ ├── ets-config.tsx ← ETS configuration form
│ │ ├── ecn-config.tsx ← ECN configuration form
│ │ ├── fdb-table.tsx ← MAC FDB data table
│ │ ├── lft-table.tsx ← IB LFT data table
│ │ └── telemetry-chart.tsx ← Time-series chart component
│ ├── hooks/
│ │ ├── use-telemetry-ws.ts ← WebSocket telemetry hook
│ │ └── use-switch-api.ts ← REST API data fetching hook
│ └── lib/
│ ├── api-client.ts ← Typed API client
│ └── types.ts ← TypeScript types (mirrors Pydantic models)
├── tests/
│ ├── conftest.py ← pytest fixtures (switch spawn, endpoint helpers)
│ ├── test_eth_forwarding.py ← Ethernet L2 forwarding tests
│ ├── test_ib_forwarding.py ← InfiniBand LID routing tests
│ ├── test_classification.py ← AUTO mode classification tests
│ ├── test_pfc.py ← PFC PAUSE generation & honoring
│ ├── test_ets.py ← ETS bandwidth scheduling
│ ├── test_ecn.py ← ECN marking behavior
│ ├── test_mgmt_api.py ← Management API JSON commands
│ ├── test_uplink.py ← Uplink port (port 8) Ethernet-only tests
│ ├── test_tor_aggregation.py ← TOR-to-TOR inter-switch link tests
│ ├── test_otel.py ← OpenTelemetry telemetry export tests
│ └── endpoint.py ← Mock endpoint process (connects to port socket)
├── deploy/
│ ├── Dockerfile ← Multi-stage build (C switch + Python + Next.js)
│ ├── docker-entrypoint.sh ← Container entrypoint (process supervisor)
│ ├── docker-compose.yml ← Full 2-rack lab + observability stack
│ ├── flatcar/
│ │ └── hicain-vswitch.service ← systemd unit for Flatcar Container Linux
│ ├── lxc/
│ │ └── hicain-vswitch.conf ← LXC container configuration
│ ├── otel-collector-config.yaml ← OTel Collector pipeline config
│ └── prometheus.yml ← Prometheus scrape config
└── qemu/ ← QEMU submodule (HiCAIN RoCE-IB-vNIC device)
14. Python Test Strategy¶
14.1 Mock Endpoint Process¶
The endpoint.py module provides a class that acts as a virtual NIC endpoint:
class MockEndpoint:
"""Connects to a switch port via UNIX SOCK_SEQPACKET socket."""
def connect(self, sock_path: str) -> None: ...
def send_eth_frame(self, dst_mac, src_mac, payload, vlan_id=None, pcp=0) -> None: ...
def send_rocev2_frame(self, dst_mac, src_mac, dst_ip, src_ip, payload, ecn=0b10) -> None: ...
def send_ib_frame(self, dlid, slid, payload) -> None: ...
def send_pfc_pause(self, priority_enable_vector, times) -> None: ...
def recv_frame(self, timeout=1.0) -> bytes: ...
def close(self) -> None: ...
14.2 Test Fixture Pattern¶
@pytest.fixture
def switch_daemon(tmp_path):
"""Spawns hicain-vswitchd with the fixed 10-port layout, yields, then tears down."""
proc = subprocess.Popen(["./switch/build/hicain-vswitchd",
"--run-dir", str(tmp_path)])
yield proc
proc.terminate()
proc.wait()
@pytest.fixture
def fabric_endpoints(switch_daemon, tmp_path):
"""Creates and connects mock endpoints to fabric ports 0–7."""
eps = []
for i in range(8):
ep = MockEndpoint()
ep.connect(str(tmp_path / f"port_{i}.sock"))
eps.append(ep)
yield eps
for ep in eps:
ep.close()
@pytest.fixture
def uplink_endpoint(switch_daemon, tmp_path):
"""Creates and connects a mock endpoint to uplink port 8."""
ep = MockEndpoint()
ep.connect(str(tmp_path / "port_8.sock"))
yield ep
ep.close()
14.3 Example Test Cases¶
def test_eth_unicast_forwarding(fabric_endpoints):
"""Endpoint 0 sends to Endpoint 1's MAC → only Endpoint 1 receives."""
def test_eth_broadcast_flood(fabric_endpoints):
"""Broadcast frame from Endpoint 0 → all other fabric endpoints receive."""
def test_ib_lid_routing(fabric_endpoints):
"""IB frame with DLID=2 → only the endpoint on the port mapped to LID 2."""
def test_auto_mode_classification(fabric_endpoints):
"""Port in AUTO mode correctly classifies Ethernet vs IB frames."""
def test_pfc_pause_halts_traffic(fabric_endpoints):
"""PFC PAUSE on priority 3 → switch stops forwarding priority-3 frames."""
def test_ecn_marking_on_congestion(fabric_endpoints):
"""When queue depth exceeds threshold, RoCEv2 packets get CE-marked."""
def test_ets_bandwidth_allocation(fabric_endpoints):
"""TC3 gets ~50% bandwidth, TC0 gets ~50% under contention."""
def test_uplink_eth_only(fabric_endpoints, uplink_endpoint):
"""Uplink port 8 forwards only Ethernet; IB frames are dropped."""
def test_uplink_no_pfc(fabric_endpoints, uplink_endpoint):
"""PFC PAUSE frames are not generated on uplink port 8."""
def test_tor_to_tor_forwarding(fabric_endpoints):
"""Two switch instances: frame from switch A port 0 reaches switch B port 0 via ISL."""
def test_otel_metrics_export(switch_daemon):
"""After traffic, OpenTelemetry metrics are exported to an OTLP collector mock."""
def test_cli_port_status(switch_daemon):
"""hicain-cli port status returns formatted table of all 10 ports."""
def test_cli_set_port_mode(switch_daemon, fabric_endpoints):
"""hicain-cli port set-mode 0 STRICT_IB changes port mode and is reflected in status."""
def test_cli_json_output(switch_daemon):
"""hicain-cli port status --json returns valid JSON matching mgmt API schema."""
def test_webui_rest_ports(switch_daemon):
"""GET /api/ports returns JSON array of all 10 ports with correct roles."""
def test_webui_rest_set_pfc(switch_daemon, fabric_endpoints):
"""PUT /api/ports/0/pfc configures PFC and GET reflects the change."""
def test_webui_websocket_telemetry(switch_daemon, fabric_endpoints):
"""WebSocket /ws/telemetry streams real-time counter updates."""
15. OpenTelemetry Observability (Port 9, Console)¶
Port 9 is the Console port, a dedicated observability channel that exports switch telemetry in OpenTelemetry (OTel) format. It does not carry data-plane traffic.
15.1 Export Protocol¶
The switch exports telemetry using OTLP (OpenTelemetry Protocol) over HTTP:
- Endpoint: Configurable via
--otel-endpointCLI flag (default:http://localhost:4318) - Protocol: OTLP/HTTP with JSON encoding (simpler to implement in C than gRPC/protobuf)
- Export interval: Configurable, default 10 seconds
15.2 Metrics Exported¶
All metrics use the hicain.switch. namespace prefix.
15.2.1 Per-Port Counters (Gauge/Counter)¶
| Metric Name | Type | Unit | Description |
|---|---|---|---|
hicain.switch.port.rx_frames |
Counter | {frames} |
Total frames received |
hicain.switch.port.tx_frames |
Counter | {frames} |
Total frames transmitted |
hicain.switch.port.rx_bytes |
Counter | By |
Total bytes received |
hicain.switch.port.tx_bytes |
Counter | By |
Total bytes transmitted |
hicain.switch.port.rx_drops |
Counter | {frames} |
Frames dropped on ingress |
hicain.switch.port.tx_drops |
Counter | {frames} |
Frames dropped on egress |
hicain.switch.port.link_state |
Gauge | 1 = UP, 0 = DOWN |
15.2.2 DCB Metrics¶
| Metric Name | Type | Unit | Description |
|---|---|---|---|
hicain.switch.port.pfc.pause_sent |
Counter | {frames} |
PFC PAUSE frames sent |
hicain.switch.port.pfc.pause_received |
Counter | {frames} |
PFC PAUSE frames received |
hicain.switch.port.pfc.paused |
Gauge | 1 = priority currently paused, 0 = flowing | |
hicain.switch.port.ecn.marked |
Counter | {packets} |
Packets marked with CE |
hicain.switch.port.queue.occupancy |
Gauge | By |
Current queue depth per priority |
hicain.switch.port.queue.buffer_size |
Gauge | By |
Max queue capacity per priority |
15.2.3 Forwarding Table Metrics¶
| Metric Name | Type | Unit | Description |
|---|---|---|---|
hicain.switch.fdb.entries |
Gauge | {entries} |
Current MAC FDB entry count |
hicain.switch.lft.entries |
Gauge | {entries} |
Current IB LFT entry count |
hicain.switch.classify_errors |
Counter | {frames} |
Frames that failed classification |
15.2.4 Resource Attributes¶
Each metric export includes these OTel resource attributes:
{
"service.name": "hicain-vswitchd",
"service.version": "0.1.0",
"hicain.switch.id": "<switch-uuid>",
"hicain.switch.role": "tor"
}
Per-metric attributes:
{
"port.id": 0,
"port.role": "fabric",
"port.mode": "AUTO",
"priority": 3 // (for per-priority metrics like PFC, queue occupancy)
}
15.3 Implementation Approach¶
The switch uses a lightweight OTLP/HTTP JSON exporter implemented in C:
- A
timerfdfires every N seconds (export interval) - The timer handler collects all port stats into an OTLP JSON metrics payload
- The payload is sent via HTTP POST to the configured OTLP endpoint
- HTTP is handled with
libcurl(widely available, simple API)
struct otel_exporter {
char endpoint[256]; /* OTLP HTTP endpoint URL */
int timer_fd; /* timerfd for periodic export */
uint32_t interval_sec; /* Export interval in seconds */
char switch_id[64]; /* Switch UUID for resource attributes */
};
15.4 Collector Integration¶
The exported OTel data can be consumed by any OpenTelemetry-compatible backend:
flowchart TB
SW["hicain-vswitchd<br/>Port 9"] -->|OTLP/HTTP JSON| COL["OTel Collector<br/>otel-collector or Grafana Alloy"]
COL --> PROM["Prometheus<br/>dashboards"]
COL --> JAEGER["Jaeger<br/>traces"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef indigo fill:#e0e7ff,stroke:#3730a3,color:#1e1b4b
class SW blue
class COL indigo
class PROM,JAEGER green
For testing, pytest tests can spin up a simple HTTP server that accepts OTLP JSON payloads and asserts on the metric values.
16. Deployment¶
16.0 Host Filesystem (Default)¶
The default deployment runs all components directly on the host filesystem, no containers required:
# Build the switch daemon
cd switch && make
# Run the switch daemon
./build/hicain-vswitchd --run-dir /var/run/hicain
# Run the CLI (separate terminal)
hicain-cli port status
# Run the Web UI (separate terminal)
hicain-webui --socket /var/run/hicain/mgmt.sock --listen 0.0.0.0:8080
Host-native deployment is the simplest option for development, single-machine labs, and debugging with gdb. All UNIX sockets, binaries, and config files live directly on the host filesystem.
16.1 Container Deployment (Optional)¶
In addition to host-native deployment, the switch stack can run in containerized environments. We support three container runtimes: Docker, Flatcar Container Linux, and LXC.
16.1.1 Container Architecture¶
Each HiCAIN TOR switch runs as a single container containing all three processes:
flowchart TB
subgraph C["Container<br/>Docker / Flatcar / LXC"]
SW["hicain-vswitchd<br/>PID 1 or supervised"]
DP["C daemon<br/>data + control plane"]
SW --- DP
SW -->|mgmt.sock| UI["hicain-webui<br/>FastAPI on :8080"]
SW -->|mgmt.sock| CLI["hicain-cli<br/>Python"]
VOL1["/var/run/hicain<br/>ports + mgmt sockets"]
VOL2["/etc/hicain<br/>configuration files"]
SW --- VOL1
SW --- VOL2
end
VOL1 --> P0["port_0.sock"]
VOL1 --> P7["port_7.sock"]
VOL1 --> P8["port_8.sock"]
SW --> OTEL[":4318<br/>OTel"]
P0 & P7 & P8 & OTEL --> HOST["Bind-mounted or shared<br/>with host or containers"]
classDef blue fill:#dbeafe,stroke:#1e40af,color:#1e3a8a
classDef green fill:#dcfce7,stroke:#166534,color:#14532d
classDef amber fill:#fef3c7,stroke:#b45309,color:#78350f
class SW,DP blue
class UI,CLI green
class VOL1,VOL2,P0,P7,P8,OTEL,HOST amber
16.1.2 UNIX Socket Sharing Across Container Boundaries¶
The switch's UNIX domain sockets must be accessible to VM processes (QEMU/RoCE-IB-vNIC) or other switch containers running on the same host. This is achieved by bind-mounting the socket directory:
# Docker: mount the socket directory as a shared volume
docker run -v /var/run/hicain-switch-a:/var/run/hicain hicain-vswitch
# The QEMU VM process (on host or in another container) accesses:
# /var/run/hicain-switch-a/port_0.sock
For TOR-to-TOR (two switch containers connected via ISL):
# Shared ISL directory between two switch containers
mkdir -p /var/run/hicain-isl
# Switch A: ports 6–7 sockets go to the shared ISL directory
docker run \
-v /var/run/hicain-a:/var/run/hicain \
-v /var/run/hicain-isl:/var/run/hicain-isl \
-e HICAIN_ISL_SOCK_DIR=/var/run/hicain-isl \
hicain-vswitch --switch-id switch-a
# Switch B: connects to the same ISL sockets
docker run \
-v /var/run/hicain-b:/var/run/hicain \
-v /var/run/hicain-isl:/var/run/hicain-isl \
-e HICAIN_ISL_SOCK_DIR=/var/run/hicain-isl \
hicain-vswitch --switch-id switch-b
16.1.3 Dockerfile¶
# Multi-stage build: compile C switch, then package with Python tools
FROM debian:bookworm-slim AS builder
RUN apt-get update && apt-get install -y \
gcc make libjansson-dev libcurl4-openssl-dev \
&& rm -rf /var/lib/apt/lists/*
COPY switch/ /build/switch/
WORKDIR /build/switch
RUN make clean && make
# Runtime image
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y \
libjansson4 libcurl4 \
python3 python3-pip python3-venv \
&& rm -rf /var/lib/apt/lists/*
# Install switch daemon
COPY --from=builder /build/switch/build/hicain-vswitchd /usr/local/bin/
# Install CLI
COPY cli/ /opt/hicain/cli/
RUN pip3 install --break-system-packages -r /opt/hicain/cli/requirements.txt
RUN ln -s /opt/hicain/cli/hicain_cli.py /usr/local/bin/hicain-cli
# Install Web UI
COPY webui/ /opt/hicain/webui/
RUN pip3 install --break-system-packages -r /opt/hicain/webui/requirements.txt
# Runtime directories
RUN mkdir -p /var/run/hicain /etc/hicain
EXPOSE 8080 4318
# Entrypoint script manages all three processes
COPY deploy/docker-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["docker-entrypoint.sh"]
16.1.4 Container Entrypoint¶
The entrypoint script starts all processes and handles graceful shutdown:
#!/bin/bash
set -e
# Start switch daemon
hicain-vswitchd \
--run-dir /var/run/hicain \
--otel-endpoint "${HICAIN_OTEL_ENDPOINT:-http://localhost:4318}" \
--switch-id "${HICAIN_SWITCH_ID:-$(hostname)}" &
SWITCH_PID=$!
# Wait for management socket to be ready
while [ ! -S /var/run/hicain/mgmt.sock ]; do sleep 0.1; done
# Start Web UI (optional, skip if HICAIN_NO_WEBUI is set)
if [ -z "$HICAIN_NO_WEBUI" ]; then
python3 /opt/hicain/webui/app.py \
--socket /var/run/hicain/mgmt.sock \
--listen "${HICAIN_WEBUI_LISTEN:-0.0.0.0:8080}" &
WEBUI_PID=$!
fi
# Handle signals for graceful shutdown
trap 'kill $SWITCH_PID $WEBUI_PID 2>/dev/null; wait' SIGTERM SIGINT
wait $SWITCH_PID
16.1.5 Flatcar Container Linux¶
Flatcar uses systemd natively. The switch is deployed as a systemd unit that pulls and runs the container image:
# /etc/systemd/system/hicain-vswitch.service
[Unit]
Description=HiCAIN Virtual TOR Switch
After=network-online.target
Requires=docker.service
[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStartPre=-/usr/bin/docker pull ghcr.io/packetfive/hicain-vswitch:latest
ExecStart=/usr/bin/docker run --rm --name hicain-vswitch \
-v /var/run/hicain:/var/run/hicain \
-v /etc/hicain:/etc/hicain:ro \
-p 8080:8080 \
-e HICAIN_SWITCH_ID=%H \
-e HICAIN_OTEL_ENDPOINT=http://otel-collector:4318 \
ghcr.io/packetfive/hicain-vswitch:latest
ExecStop=/usr/bin/docker stop hicain-vswitch
[Install]
WantedBy=multi-user.target
Flatcar's immutable filesystem makes container-based deployment the natural choice. Configuration goes in /etc/hicain/ (writable) and is bind-mounted into the container.
16.1.6 LXC¶
For LXC, the switch runs as a system container providing a full lightweight Linux environment:
# Create and configure the LXC container
lxc-create -n hicain-vswitch -t download -- -d debian -r bookworm -a amd64
# Share socket directory with host
echo "lxc.mount.entry = /var/run/hicain var/run/hicain none bind,create=dir 0 0" \
>> /var/lib/lxc/hicain-vswitch/config
# Start the container
lxc-start -n hicain-vswitch
# Inside the container: install and run the switch
lxc-attach -n hicain-vswitch -- /usr/local/bin/hicain-vswitchd --run-dir /var/run/hicain
LXC shares the host kernel, so UNIX domain sockets work seamlessly via bind mounts, no network namespace translation needed.
16.1.7 Environment Variables¶
All container configuration is driven by environment variables (12-factor style):
| Variable | Default | Description |
|---|---|---|
HICAIN_SWITCH_ID |
$(hostname) |
Unique switch identifier (used in OTel resource attributes) |
HICAIN_OTEL_ENDPOINT |
http://localhost:4318 |
OTLP collector endpoint |
HICAIN_WEBUI_LISTEN |
0.0.0.0:8080 |
Web UI listen address |
HICAIN_NO_WEBUI |
(unset) | Set to disable Web UI process |
HICAIN_ISL_SOCK_DIR |
(unset) | Override directory for ISL port sockets (TOR-to-TOR) |
HICAIN_RUN_DIR |
/var/run/hicain |
Runtime directory for sockets |
HICAIN_LOG_LEVEL |
info |
Log verbosity: debug, info, warn, error |
16.1.8 Docker Compose, Full Lab Topology¶
A complete two-rack lab with observability stack:
# docker-compose.yml
version: "3.8"
volumes:
switch-a-sockets:
switch-b-sockets:
isl-sockets:
services:
# --- Switches ---
switch-a:
build: .
environment:
HICAIN_SWITCH_ID: switch-a
HICAIN_OTEL_ENDPOINT: http://otel-collector:4318
HICAIN_ISL_SOCK_DIR: /var/run/hicain-isl
volumes:
- switch-a-sockets:/var/run/hicain
- isl-sockets:/var/run/hicain-isl
ports:
- "8080:8080" # Web UI Switch A
switch-b:
build: .
environment:
HICAIN_SWITCH_ID: switch-b
HICAIN_OTEL_ENDPOINT: http://otel-collector:4318
HICAIN_ISL_SOCK_DIR: /var/run/hicain-isl
volumes:
- switch-b-sockets:/var/run/hicain
- isl-sockets:/var/run/hicain-isl
ports:
- "8081:8080" # Web UI Switch B
# --- Observability ---
otel-collector:
image: otel/opentelemetry-collector:latest
ports:
- "4318:4318" # OTLP HTTP
volumes:
- ./deploy/otel-collector-config.yaml:/etc/otelcol/config.yaml
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./deploy/prometheus.yml:/etc/prometheus/prometheus.yml
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
GF_SECURITY_ADMIN_PASSWORD: hicain
17. Build System¶
The switch daemon will use a simple Makefile with GNU C flags:
CC = gcc
CFLAGS = -std=gnu11 -Wall -Wextra -Wpedantic -g -O0
LDFLAGS = -ljansson -lcurl # JSON parsing + HTTP for OTel export
-std=gnu11, GNU C11, consistent with Linux kernel style-g -O0, debug symbols, no optimization (educational tool)-ljansson, Jansson for JSON parsing (lightweight C JSON library)-lcurl, libcurl for OTLP/HTTP metric export
18. Open Design Questions¶
- FDB aging: Should the MAC FDB age out entries after a timeout, or only flush on link-down? (Suggest: configurable timer, default 300s)
- Buffer model: Should per-priority queue sizes be configurable at startup or only via the management API? (Suggest: CLI args for defaults, management API for runtime changes)
- Logging:
syslog,stderr, or a custom log file? (Suggest:stderrfor educational visibility, with a--log-levelflag) - IB Subnet Manager emulation: Should the switch include a minimal SM that assigns LIDs, or is static LFT configuration via management API sufficient for v1? (Suggest: static LFT for v1)
- ISL LAG: Should inter-switch link aggregation (ports 6, 7 bonded) be in v1 scope? (Suggest: defer to v2; use independent ISL links with static entries for v1)
- OTel traces: Should the switch emit per-frame traces (span per frame forwarding decision) in addition to metrics? (Suggest: metrics-only for v1, traces for v2 debugging mode)
- Web UI authentication: Should the Web UI require login credentials? (Suggest: no auth for v1, it's a local educational tool. Add optional basic auth for v2)
- Starlark DSL for lab orchestration (v2): Use Python Starlark as a declarative DSL for defining multi-switch lab topologies (switches, VMs, ISLs, DCB policies). Would generate Docker Compose + JSON management commands from a single
.starfile. (Suggest: defer to v2, v1 uses direct CLI/API configuration for single-switch setups)
Resolved Decisions¶
| Decision | Choice |
|---|---|
| Python CLI framework | Typer (with asyncio, Pydantic, Rich) |
| Python tooling | Poetry (packaging), Ruff (linting), mypy (type checking) |
| Web UI frontend | Next.js + React + TypeScript |
| Web UI charting | Recharts (telemetry), React Flow (topology) |
| Web UI styling | Tailwind CSS + shadcn/ui |
| Documentation | MkDocs Material |
| Deployment default | Host filesystem (containers are optional) |