Hi! I'm Neil McGlohon

IBM AIU NorthPole Rack-Scale LLM Inference

I was a primary contributor to the vertical integration of the IBM AIU NorthPole Chip into an end-to-end LLM inference system comprising 288 NorthPole accelerator cards, a high-performance runtime stack, and a containerized inference pipeline.

The research prototype system delivers 115 peta-ops at 4-bit precision and 3.7 PB/s of memory bandwidth across eighteen 2U servers. The system only consumes 30 kW of power, enabling it to be deployed in existing data center (cloud or on-prem), without requiring exotic communication fabrics, custom hardware integration, liquid cooling, or facility power upgrades.

The modular, scalable, and reconfigurable system can run 3 simultaneous instances of the IBM Granite-3.3-8b-instruct model at 2,048 context length with 28 simultaneous users at a per-user inter-token latency of 2.8 ms -- for a total of 30,000 tokens/sec. The same system can run 18 instances of a 3-billion-parameter model at the same context length and number of users, achieving an inter-token latency of 1 ms -- for a total of 504,000 tokens/sec.

The work for this project was summarized in a pre-print article on ArXiV: https://arxiv.org/abs/2511.15950

Close Window

IBM AIU NorthPole Server

I wrote a bespoke LLM inference engine for the IBM AIU NorthPole chip. An initial demonstration of this engine featured the deployment of the IBM Granite 3B LLM model on a compact 2U research prototype system using the IBM AIU NorthPole inference chip. Early benchmarking results showcased an astounding 28,356 tokens/sec of system throughput and sub-1ms/token (per-user) latency. NorthPole is optimized for the two conflicting objectives of energy-efficiency and low latency. In the regime of low-latency, NorthPole (in 12nm) provides 72.7x better energy efficiency (tokens/second/W) versus a state-of-the-art 4nm GPU. In the regime of high-energy efficiency, NorthPole (in 12nm) provides 46.9x better latency (ms/token) versus a 5nm GPU.
The work for this project was summarized in a paper published at IEEE HPEC 2024. https://ieeexplore.ieee.org/document/10938418

Close Window

IBM AIU NorthPole Edge Inference

Commonly, high-performance AI runs primarily in the data center. While it makes sense for training to stay there, there's significant value in inference tasks operating in edge environments, minimizing transmission costs and latency for in-situ mission critical scenarios where microseconds matter.

To enable inference outside the data center, users need AI accelerators with both high performance and high energy efficiency, embodied in a form factor optimized for deployment at the edge.

The IBM AIU NorthPole chip stands to fulfill that need. It is not only capable of inference tasks with orders of magnitude better energy efficiency than that of comparable silicon-process GPU systems, but it is able to simultaneously provide additional orders of magnitude improved throughput and latency metrics.

I joined the IBM AIU NorthPole team after they had started writing the Science journal paper and missed the author cutoff but contributed to the work and the unpublished evolutions that followed. https://www.science.org/doi/full/10.1126/science.adh1174

Close Window

CODES HPC Network Simulator

The Co-Design of Exascale Storage Architectures (CODES) project is a parallel discrete event simulation framework that allows for massive scale simulation of high performance computing interconnects, workloads, and technologies. I am the primary maintainer of this project's repository.
github.com/codes-org/codes

Close Window

ROSS Simulator

The Rensselaer Optimistic Simulation System (ROSS) is a Parallel Discrete Event Simulation engine designed with many-processor scalability in mind. It offers parallel execution of developed models in an optimistic fashion, allowing each physical processor to process simulation events without any forced periodic inter-core synchronization yielding significant performance boosts over other parallel execution modes. I am a contributor to this project and repository.
github.com/ross-org/ross

Close Window

Interconnect Topology Research

In the process of writing my Ph.D dissertation, I've spent years studying the history and the state of the art of high performance computing communication interconnects. These networks of switches and network interface cards are the backbone of every supercomputer. The power, speed, efficiency and cost of a supercomputer is incredibly dependent on a suitable choice in network topology.

Close Window

Congestion Control

My Ph.D dissertation focuses on the analysis of congestion in high performance computing communication interconnects and methods for both treating it once it occurs as well as preventing it from occuring in the first place. Passive prevention can take the form of Quality of Service techniques. Active abatement measures are called Congestion Management: a class of strategies employed to actively treat congestion as quickly and efficiently as possible without negatively affecting other unrelated operators on the network.

Close Window