A December 17, 2024 benchmark published on Hugging Face reported substantially higher language-model throughput on Google Cloud’s fifth-generation Intel Xeon C4 instances than on older N2 machines. The authors reported 10–24× throughput for embeddings and 2.3–3.6× for text generation—but those figures are historical, vendor-involved results, not independently verified measurements. 1
The post’s authors include Matrix Yao and Ke Ding, identified as Intel contributors, alongside Ilyas Moutawwakil. The supplied research review did not rerun the benchmarks and found no independent reproduction. ScholarBro’s secondary coverage links back to the original publication rather than providing another set of measurements. 1 2
The results provide a reason to investigate CPU-only language-model workloads. They do not establish current cloud savings, superiority over GPUs or production-agent readiness. 1
What the benchmark reported
The authors compared n2-standard-96 instances using Ice Lake processors with c4-standard-96 instances using Emerald Rapids processors. Their reported throughput ratios varied across batch sizes: 1
WhereIsAI/UAE-Large-V1
1–128
10–24×
Text generation
meta-llama/Llama-3.2-3B
1–64
2.3–3.6×
These ranges describe the authors’ selected workloads and batch sweeps, not a general performance multiplier for all language models. 1
Using an assumption that C4 hourly pricing was approximately 1.3× higher, the authors also claimed 7–19× embedding and 1.7–2.9× generation advantages under their TCO framing. The retrieved post did not establish a pricing region and date, and the review did not verify current prices. Those estimates should therefore be treated as historical, assumption-dependent cost-efficiency claims rather than present-day savings. 1
Configuration evidence helps—but does not complete reproduction
The referenced optimum-benchmark configuration files, pinned to commit d58bb2582b872c25ab476fece19d4fa78e190673, specify CPU execution, the IPEX backend, BF16 precision, model export and real weights. The embedding configuration uses 128-token inputs; the generation configuration specifies 256 input tokens and exactly 32 new tokens. 3 4
Those files are starting configurations, not final experiment manifests. Both default to batch size 1 and retain their original BERT or TinyLlama model identifiers rather than the models named in the published results. Neither supplies the complete batch sweep. The generation file specifies 10 warmup runs and 10 iterations, but that does not demonstrate repeated testing across separately provisioned virtual machines. 3 4
The supplied review also found no absolute result tables or downloadable run logs in the retrieved original post. Its separate 13× batch-scaling claim remains difficult to interpret because the denominator and numerical latency bound are unclear. Fully resolved configurations, sweep commands and measurement logs would be needed to close these reproduction gaps. 1 3 4
The comparison does not isolate the source of the gains
Equal vCPU counts do not make the two machines identical. Google’s specifications list 360 GB of memory for c4-standard-96 and 384 GB for n2-standard-96. Google also defines a vCPU as a hardware thread, with two threads per core for these machines: 96 vCPUs should not be described as 96 physical cores. 5
Intel’s documentation identifies Advanced Matrix Extensions, or AMX, as supporting BF16 and INT8 and as having been introduced with fourth-generation Xeon. The benchmark crosses from third-generation Ice Lake to fifth-generation Emerald Rapids; it is not an incremental fourth-to-fifth-generation comparison. AMX is a plausible contributor, but the retrieved study provides no AMX-disabled control to isolate its contribution. 6 1
Keeping BF16 constant controls precision, but does not establish that each platform used its fastest suitable precision and runtime combination. A platform-tuned comparison would help assess how much the results depend on those choices. 3 4
C4 now covers newer hardware, too
Google announced general availability of C4 with fifth-generation Xeon on August 19, 2024. Its release notes subsequently recorded Xeon 6 C4 availability on July 30, 2025, and at Next ’26 the company announced expanded Xeon 6 support across all C4 shapes. The original benchmark should not be presented as a new 2026 announcement or a test of Google Cloud’s latest Xeon generation. 7 8 9
Google’s documentation now discusses both Emerald Rapids and Granite Rapids within C4. A present-day reproduction therefore needs to record the actual CPU platform, not merely the C4 family name. 5
An October 16, 2025 Intel/Hugging Face follow-up reported 1.4–1.7× throughput per vCPU for Xeon 6 C4 against fourth-generation Xeon C3. That study used GPT-OSS-120B in BF16 with 1,024-token inputs and outputs. Different models, sequence lengths, hardware and normalization prevent direct comparison with the 2024 multipliers. It is another CPU-versus-CPU test, not a reproduction or a GPU comparison. 10
The follow-up has its own qualifications. Its hardware table lists 172 C3 vCPUs while its setup instructions specify 176, leaving the normalization denominator unresolved. Its claimed 1.7× cost-efficiency advantage would imply approximately 41% lower cost for fixed output—calculated as 1 − 1/1.7—not a 70% cost reduction. 10
What remains to be tested
The original study reported no GPU comparison, model-quality evaluation or complete-agent benchmark. Its component-level throughput results cannot establish those broader outcomes. 1
For teams considering a rerun, the outstanding questions are practical:
- Can the authors supply final configurations, every batch-sweep command and repeated-run measurements? 3 4
- How do platform-optimized runtimes and precision choices, and AMX-enabled versus disabled runs, change the comparison? 6 3
- Do the gains persist with longer prompts and outputs, realistic concurrent arrivals and explicit latency targets? 4
- Which CPU generation will actually be provisioned, and what region-specific prices and discounts apply? 5
Until those questions are answered, the benchmark is best used as historical evidence motivating fresh tests—not as a current procurement verdict.
Research scope: The supplied review covered source verification and methodology through October 12, 2026. Benchmarks were not rerun, and independent reproduction was not located.