ML.NET vs Python on Clinical Data: Three Gotchas From the Setup

I'm benchmarking ML.NET against scikit-learn, XGBoost and LightGBM on open clinical datasets. The question is whether you can skip the Python service when a model has to live inside a .NET app. This post isn't about the results. It's about three things from the setup that bit me, which you will probably hit too.

ML.NET LightGBM does not run on Apple Silicon

Microsoft.ML.LightGbm 5.0.1 ships no native library for osx-arm64. On an M-series Mac, logistic regression and FastTree work and LightGBM fails. I tested on macOS 14.5 with .NET 8.0.4. Run it on Windows or Linux x64 instead. I use two Windows machines and GitHub Actions runners. Don't hand-build a different LightGBM binary to get around it, because a version mismatch will quietly change your numbers.

ML.NET logistic regression gives different predictions on different machines

Same data, same folds, same seed, two Windows machines, different predictions. The cause was L-BFGS threading. Pin it to one thread:

new LbfgsLogisticRegressionBinaryTrainer.Options { NumberOfThreads = 1 }

With that, both machines produced bit-identical predictions on all four of my small datasets.

ONNX Runtime flipped a prediction at exactly 0.5

I train in Python, export to ONNX, and score from C#. A random forest of 200 fully grown trees returns exactly 0.5 when 100 trees vote each way. ONNX Runtime averages in float32 and returned 0.49999967 for that row. The difference is 3.3e-7, but with a 0.5 threshold it becomes a different decision. If you check fidelity by counting flipped decisions, you will flag these. Look at the largest probability difference first and treat flips right at the threshold as rounding ties.

What "matched" means here

Both sides read the same frozen feature file and the same fold CSV. Tuning folds come from a splitmix64 hash of seed, repeat, fold and row id, implemented separately in C# and Python and checked against each other. Tree models get 200 trees, no early stopping and one thread.

It still isn't an identical comparison. XGBoost grows trees depth-wise while the others grow leaf-wise, and ML.NET and scikit-learn scale the L2 penalty differently, so an L2 value in one is not proven equal to a C value in the other. I'll say so plainly in the write-up.

What's next

The benchmark compares AUROC, calibration, latency, cold start and memory on open clinical datasets. I'll write up the full study as a preprint and post a follow-up here once it's public. If you run ML.NET in production and have a question I should answer, email me.