BeoShock HPC System Overview

This document describes a dedicated high-performance compute (HPC) node provisioned within the institutional faculty condominium (condo) queue. The system is allocated for the exclusive use of a single faculty member and their research group, while being physically and operationally integrated into the shared cluster environment. The node has been designed to support GPU-accelerated scientific computing, large-scale Monte Carlo workloads, and hybrid MPI/OpenMP/CUDA applications.

This resource is immediately available and does not require additional acquisition or development.

The configuration reflects a balance between high core-count CPUs, multiple large-memory GPUs, and NUMA-aware system architecture, enabling both tightly coupled multi-GPU jobs and concurrent independent GPU workloads.

The compute node compute202515 is a dual-socket AMD EPYC Gen4 system equipped with two AMD EPYC 9565 processors, providing a total of 144 physical CPU cores (72 cores per socket, single thread per core) across two NUMA domains. The node includes four NVIDIA RTX Pro 6000 GPUs based on the Blackwell architecture, each with approximately 96 GB of ECC-protected device memory, for an aggregate GPU memory capacity of approximately 384 GB. GPUs are connected via PCIe and are evenly distributed across the two NUMA domains, with two GPUs local to each CPU socket. The software environment includes NVIDIA Driver version 575.51.03 with CUDA 12.9, supporting modern vector and mixed-precision execution on both CPUs and GPUs.

The node provides approximately 5 TB of node-local NVMe storage, mounted at /mnt/compute_data, which is intended for temporary job data, scratch usage, and data staging during active computations. Additional local storage is used for operating system and system software installation. Persistent user data are not stored on the node and are instead provided through shared, facility-managed storage systems accessible from all compute nodes.

Compute Node Type

• Node classification: Faculty-owned condo node
• Queue: Faculty condominium queue (restricted access)
• Usage model: Exclusive to owning faculty member and approved group users

The node is fully managed by central IT / HPC operations, including operating system provisioning, scheduler integration, monitoring, and security patching, while computational capacity is reserved for the faculty owner.

• Processor model: AMD EPYC 9565 (Zen 4 / Genoa)
• Sockets: 2
• Cores per socket: 72
• Total physical cores: 144
• Threads per core: 1 (SMT disabled)
• NUMA domains: 2 (one per socket)

The absence of simultaneous multithreading simplifies performance predictability and avoids resource contention, which is advantageous for MPI and hybrid CPU/GPU workloads.

CPU Features

• AVX2 and AVX‑512 instruction sets
• AVX‑512 BF16 and VNNI support
• Large on‑chip cache hierarchy (768 MB aggregate L3)

These features support efficient vectorized computation, mixed‑precision workflows, and memory‑intensive scientific applications.

• GPU model: NVIDIA RTX Pro 6000 (Blackwell architecture)
• Number of GPUs: 4 per node
• Memory per GPU: ~96 GB ECC VRAM
• Aggregate GPU memory: ~384 GB
• Maximum power per GPU: 600 W
• MIG: Disabled

Each GPU is available as a full device to user jobs, enabling large on‑device problem sizes without memory partitioning.

GPU Software Environment

• NVIDIA driver version: 575.51.03
• CUDA version: 12.9

The software stack supports CUDA, CUDA‑aware MPI, and modern GPU programming models such as Kokkos (CUDA backend).

The node exhibits a clean and symmetric NUMA layout:

• NUMA node 0: GPU0 and GPU1
• NUMA node 1: GPU2 and GPU3

All GPU–GPU connectivity occurs via PCIe within a NUMA node. No NVLink interconnect is present. This topology favors explicit GPU ownership per MPI rank and NUMA‑local process placement.

High‑speed network interfaces (Mellanox NICs) are similarly distributed, with each socket hosting network devices in close PCIe proximity, supporting efficient MPI communication and RDMA operations.

The node is optimized for the following execution model:

• Four MPI ranks per node, each bound to a single GPU
• NUMA‑aware pinning, ensuring each rank uses CPU cores local to its GPU
• Optional OpenMP threading within each rank (typically 8–16 threads)

This configuration minimizes cross‑socket memory traffic and maximizes sustained GPU utilization.

This condo node is well suited for:

• GPU‑accelerated Monte Carlo simulations
• Parameter scans and ensemble workflows
• Hybrid MPI + CUDA or MPI + Kokkos applications
• Large‑memory GPU workloads requiring tens of gigabytes per device

The hardware supports both tightly coupled multi‑GPU jobs and simultaneous independent GPU workloads, depending on user preference and scheduler policy.

As a faculty condo queue resource, this node:

• Expands the institution’s aggregate GPU capacity
• Provides guaranteed access for faculty‑owned research programs
• Operates under the same scheduler, monitoring, and security framework as general‑access cluster resources

This model enables customized, high‑end computational capability for individual research programs while maintaining operational consistency across the HPC facility.

This faculty condo compute node represents a high-end, GPU-dense resource tailored for modern scientific computing. Its combination of 144 Zen 4 CPU cores, four large-memory Blackwell GPUs, and a clean NUMA-aware architecture makes it an effective and flexible platform for advanced research workloads within the shared facility environment.

To view CPU model, core counts, sockets, NUMA layout, and supported instruction sets:

1     lscpu 2

This command reports processor model, number of sockets, cores per socket, NUMA node layout, cache sizes, and supported instruction extensions.

To list available GPUs and basic status information:

1     nvidia-smi 2

This reports GPU models, memory capacity, driver version, CUDA version, and current utilization.

To examine GPU-to-GPU and GPU-to-CPU topology (NUMA locality and PCIe connectivity):

1     nvidia-smi topo -m 2

To display mounted filesystems and their sizes:

1     df -h 2

Local node storage typically appears as devices mounted on directories such as /, /scratch, /local, or /mnt, while shared facility storage is usually mounted via network filesystems (e.g., NFS, Lustre, BeeGFS).

To identify the underlying block devices and local disks:

1     lsblk 2

To check the size of shared memory (RAM-backed temporary storage):

1     df -h /dev/shm 2

This space is useful for I/O-intensive workloads but is not persistent.

For facilities documentation, users should avoid assuming hardware specifications across nodes and instead verify configuration directly on representative systems using the commands above. Reported values reflect the actual, installed hardware and software environment at the time of execution.