Skip to content
NetProjects
Menu

Insights

H2O-Lightning-4B on a Mac: a calibrated decision model that knows when it isn't sure

Paris Michalas, Partner, NetProjects (Internet Projects Ltd)

Many sites need a model that answers one narrow question and says how sure it is. Is this claim supported by the evidence? Which priority is this ticket? Does this record match? A chat model can answer those questions, but its stated confidence is rarely something you can rely on. H2O.ai's H2O-Lightning-4B is built for exactly this job. It reads a short record and a question, and returns a probability for each possible answer in a single pass.

Its official deployment runs on vLLM, which needs an NVIDIA GPU. We wanted it on a laptop, so we ported the serving layer to Apple Silicon with MLX. The official shim runs unchanged. We replaced only the part that needed NVIDIA. When we checked Hugging Face on 12 October 2026, this was the first public MLX build we could find.

What we built

  • An MLX serving facade. It answers the five vLLM calls the official shim makes, using one MLX forward pass per question. Nothing is sampled. The hidden state at the answer position goes through the output head in float32, as the model's configuration requires.
  • An optional front door. A small server adds an API key and a simple single-question route, for applications that send one question at a time.
  • Quantised builds. An 8-bit build (4.8 GB on disk, 6.3 GB peak) and a 4-bit build (2.9 GB on disk, 4.5 GB peak), each with a model card.

Does the port match the original?

We tested on the public JevBench set (231 items, CLI commit bb05a33) on a MacBook Pro M5 Max. H2O's model card gives the original build 205 out of 231, and the full-precision (bf16) weights on MLX score 204.

  • 8-bit: 205 out of 231, the same score as H2O's model card. 229 of 231 answers are identical to the bf16 model on MLX, and the probabilities differ by 0.006 on average. This is the build we recommend.
  • 4-bit: 207 out of 231, which is within noise. It gives the same answer as the bf16 model on 220 of 231 items, and its probabilities drift more (0.039 on average). It is the small-memory option, looser calibration.
  • Speed: in our measurements, the 8-bit build takes about 0.4 seconds to read a 2,000-token record and answer one question. Items of about 650 tokens take about 0.06 seconds.

Is the confidence worth relying on?

Accuracy is only half of the story for a decision model. The useful signal is the confidence. We checked all three MLX builds on 400 items from the public VitaminC test split (200 supported, 133 refuted, 67 not enough info), with the evidence sentence buried in about 650 words of other text.

  • bf16: 81.2% accuracy, Brier score 0.135, calibration error (ECE) 0.057, AUROC 0.891.
  • 8-bit: 81.2% accuracy, Brier score 0.134, calibration error (ECE) 0.055, AUROC 0.892.
  • 4-bit: 82.8% accuracy, Brier score 0.134, calibration error (ECE) 0.068, AUROC 0.888.

Lower is better for the Brier score and calibration error. The accuracy differences are within noise (about ±4 points at 400 items). The calibration is good and holds across builds, with the 4-bit build a little looser. With the 8-bit build, accepting only answers at confidence 0.9 or higher covered 58% of the items at 90% accuracy, so the confidence tells you which answers to double-check.

We also looked at H2O (8-bit) on our own set of 512 claims from a travel-content site we run. It is our own set, one site, not a public benchmark, and it leans towards hard, low-confidence claims. When H2O's confidence was 0.80 or higher, it was right 90.8% of the time (262 claims); below that, it was right 54.0% of the time (250 claims).

“A decision model is only useful if its confidence means something. H2O's does: when it was confident it was right about nine times in ten, and when it wasn't sure, it said so.”

— Paris Michalas, Partner, NetProjects (Internet Projects Ltd)

Try it

The code is at https://github.com/internet-projects/h2o-lightning-mlx. Its setup script downloads the official H2O repository, converts the weights to 8-bit, and runs the shim's own start-up checks. The README has the full benchmark commands, so you can reproduce the public benchmark numbers above.

What we are releasing, and what stays private

Public: the serving code, the model cards, and the benchmark scripts. Private: our product harness, our prompts and our data.

On 12 October 2026 Paris also shared the build in the H2O.ai community discussion for the model and with the MLX Community on Hugging Face:

Credits

H2O-Lightning-4B is by H2O.ai (Apache-2.0), built on Qwen3.5-4B by the Qwen team (Apache-2.0). The MLX port is by NetProjects (Internet Projects Ltd) and is Apache-2.0 licensed. NetProjects is not affiliated with H2O.ai. Thanks also to the MLX and mlx-vlm projects, and to JevBench.

Paris Michalas is a partner at NetProjects, the trading name of Internet Projects Ltd. This article was drafted with AI assistance, then edited and approved by people on our team. You can read more about how we use AI at https://netprojects.com/standards/ai.