Designing and Building a High-Performance 6-GPU Local AI Workstation: Architecture Insights and Feedback
Introduction
In the rapidly evolving landscape of artificial intelligence, having a robust local infrastructure can significantly enhance data processing, analytics, and automation capabilities. Recently, I embarked on constructing a high-end AI workstation equipped with six GPUs, built on a Threadripper PRO platform, tailored to facilitate internal analytics and automation tasks. While I am relatively new to high-performance hardware assembly, extensive research over the past year guided this effort. Here, I share the architecture details, objectives, and seek expert feedback on the system’s design and future scalability.
Project Overview and Objectives
The primary goal of this build is to ingest approximately one year’s worth of diverse internal business data, including emails, instant messages, attachments, call transcripts, and database exports. The system aims to support:
- Development of a vector and graph retrieval layer for advanced data querying
- Execution of reasoning models locally to analyze processes, detect patterns, and automate workflows
- Streamlining manual operations through AI-driven tools
Though this setup may seem overbuilt for the dataset size, I prioritized future-proofing by designing a scalable infrastructure capable of handling expanded datasets and more complex workloads.
Hardware Architecture
Here is the detailed hardware configuration of the workstation as of February 2026:
System Case & Environment
– Open-air rig setup to facilitate cooling and hardware accessibility
– Operating System: Ubuntu 24.04 LTS Desktop, ensuring stability and AI-friendly compatibility
Motherboard
– ASUS WRX90E-SAGE Pro WS SE (AMD sTR5, EEB form factor)
Processor
– AMD Ryzen Threadripper PRO 9955WX (16 cores, 4.5 GHz boost, Shimada Peak architecture)
Memory
– 256GB DDR5 ECC Registered RAM (8 x 32GB Kingston FURY Renegade Pro 5600 MHz modules)
Storage
– Two Samsung 990 PRO NVMe SSDs (4TB each) for active data and operations
– One Samsung 9100 PRO NVMe SSD (8TB) with PCIe Gen 5 support and heatsink for large datasets
Power Supply
– Dual PSUs for redundancy:
– SilverStone Hela 2500Rz (2500W, Platinum rated)
– MSI MEG Ai1600T (1600W, Titanium rated)
– Power adapters and daisy-chaining solutions ensure stable power delivery
Cooling and Airflow
– Thermaltake WAir CPU air cooler
– External Arctic P12 PWM PST fans mounted to GPU heatsinks
– Arctic 10-port PWM fan hub with SATA power for managing multiple fans
Uninterruptible Power Supply (UPS)
– CyberPower PR3000LCD, 3000VA capacity, with sinewave output for uninterrupted operation
Graphics Processing Units (GPUs)
– Main GPUs:
– 2 x PNY RTX 6000 Pro Blackwell (professional-grade)
– 3 x RTX 3090 Ti (FE and EVGA variants)
– Additional GPU:
– Dell 3090 (small form factor)
– Future Expansion:
– Potentially adding RTX 6000 Pro Maxwell cards
Connectivity and Riser Cables
– PCIe 5.0 riser cables (30cm & 60cm) from LINKUP for flexible GPU placement and optimal bandwidth
Operational Milestones
A significant achievement was successfully running all six GPUs concurrently with three open models without encountering stability issues—a notable milestone in my setup journey.
Budget and Cost Considerations
Initially, I estimated the total cost at approximately $50,000 USD. After recalculation, the actual expenditure totals around $38,000 USD. I appreciate the community in pointing out this correction and welcome any insights on optimizing costs in similar builds.
Key Questions and Architectural Reflection
As I refine this infrastructure, I am eager to hear expert opinions on the following:
-
Bottleneck Identification: With this scale, what is typically the first resource to become a limiting factor—VRAM, PCIe bandwidth, CPU orchestration, or another component?
-
GPU Heterogeneity: Does mixing GPU types pose long-term operational challenges, or can workloads be managed effectively through careful assignment?
-
Scalability and Diminishing Returns: When running multiple models concurrently, do performance gains plateau after a certain number of models or GPUs?
-
Data Layer Architecture: For internal document and database analysis, is investing early in a full graph database justified, or does this risk overbuilding initial infrastructure?
-
System Design Philosophy: Should I prioritize building a single, high-powered machine or multiple smaller nodes for scaling?
-
Common Pitfalls: What are frequent mistakes to avoid when designing larger on-premise AI systems for internal enterprise use?
Closing Remarks
As I continue learning and developing this infrastructure, constructive critiques and shared experiences are invaluable. I look forward to engaging discussions that can help optimize this system’s architecture and guide future enhancements.
Thank you for your insights and support.
