BESTEST Validation of Aladdin’s Building Energy Simulation

← Back to Aladdin

By Charles Xie ✉

When a piece of software tells you that a house will use 4,700 kilowatt-hours of heating next year, how do you know whether to believe it? You cannot look the answer up — the whole point of simulation is to predict something that has not happened yet. You cannot eyeball it, either: a building's energy balance is a tangle of sunlight, conduction, air leakage, thermal mass, and thermostat logic, all pushing on each other hour by hour. The building simulation community confronted this trust problem decades ago, and its answer has become one of the quiet institutions of the field: a plain gray box in Denver, Colorado, that every serious simulation program in the world has been asked to model. It is called BESTEST — the Building Energy Simulation Test — and Aladdin has now taken the test.

A standard made of buildings that were never built

BESTEST was developed at the U.S. National Renewable Energy Laboratory in the 1990s under the International Energy Agency, and was later adopted as ANSI/ASHRAE Standard 140, the standard method of test for building energy software. Its central idea is elegant: instead of comparing a simulation against a messy real building — where the occupants, the weather station, and the utility meter all add their own noise — it defines a set of idealized buildings so precisely that nothing is left to interpretation, and asks every program to simulate exactly the same thing. The base case, Case 600, is an 8 × 6 × 2.7 meter single-zone box with twelve square meters of south-facing glass, specified down to the conductivity of each layer of drywall, sitting in Denver weather. It has never been built. It does not need to be. Its job is to be unambiguous.

From that base case, the standard changes exactly one thing at a time: add a south overhang (Case 610), move the windows to the east and west walls (620), shade them with fins (630), let the thermostat set back at night (640), ventilate with a night fan instead of air conditioning (650) — and then rebuild the whole series with heavy concrete walls and a concrete slab floor (the 900 series). There are also “free-float” cases with no heating or cooling at all, where the building simply rides the weather and the simulator must predict how hot and cold the interior gets. Because each case isolates one physical mechanism — solar gain, shading, thermal mass, setback recovery, ventilation — the pattern of agreement and disagreement across cases is diagnostic: it tells you not just whether a program deviates, but which piece of physics is responsible.

The measuring stick is the community itself. The current edition of the standard publishes example results from six established programs — EnergyPlus, TRNSYS, ESP-r, DeST, CSE, and BSIMAC — each maintained by a different institution on a different continent. Even these veterans do not agree exactly; their spread on some monthly results exceeds twenty percent, which is a humbling reminder of how hard this physics is. The standard is careful to say that the ranges “are not to be interpreted as acceptance criteria” — there is no official pass or fail. What the ranges give you is the honest question: does your program behave like the programs the field trusts?

How Aladdin takes the test

Aladdin's building energy tools include a transient (dynamic) thermal simulation: a resistor–capacitor network of the building derived from the ISO 13790 standard, extended with a separate thermal node for the floor slab — because that is where the sun patch lands — and driven hour by hour through a full year of measured typical-year (TMY3) weather. Sunlight is traced against the actual 3D geometry, so an overhang shades a window in the model exactly the way it shades it on screen. For these benchmarks we ran every case as a continuous 365-day simulation of the same Denver weather the reference programs used, with the air conditioner's coefficient of performance set to one so that the reported numbers are thermal loads, directly comparable to the published tables.

All twenty-seven cases — the core conditioned suite, the window-technology variants (low-e argon and single-pane glazing, each with its own WINDOW 7 angular optics), the increased-insulation and zero-deadband variants, six free-floating buildings, and the two-zone sunspace (Case 960) — are built into Aladdin under Benchmarks → Building Energy, authored to the letter of the specification: the exact layer-by-layer wall constructions, the 0.41 air changes per hour of infiltration (adjusted for Denver's altitude, as the standard instructs), the 200 watts of continuous internal gains, the 0.6 solar absorptance, even the glazing's angle-by-angle optical data from the LBNL WINDOW report that ships with the standard. This matters for a reason beyond bookkeeping: it makes the whole exercise reproducible. Anyone — a skeptical colleague, a student, a reviewer — can open the same cases in a browser and run them.

Twenty-seven is not all of them: the standard’s thermal-fabric section defines fifty cases. The other twenty-three form an in-depth diagnostic series — a solid conduction box with the physics added back one mechanism at a time, plus variants that prescribe unusual surface properties (infrared emittances of 0.1, interior solar absorptances of 0.1, fixed film coefficients). They exist to localize disagreements between programs, not to represent buildings, and several deliberately vary quantities Aladdin fixes at their standard values to keep its interface simple. We skip them for now, openly: they would become worthwhile the day a regression needs hunting, not as a scorecard. Standard 140 also contains two further chapters entirely — analytical and comparative tests for space-cooling and space-heating equipment — which exercise HVAC hardware models rather than the building fabric, and which would be their own validation campaign.

The sunspace case deserves a word, because it exercises something new: Case 960 is a two-zone building — a conditioned back room behind a free-floating, heavily glazed sun room, separated by a massive concrete common wall. In Aladdin it is modeled the way users naturally compose complex houses: as two structures whose shared surfaces are declared interior. The simulation then treats them as coupled thermal zones, exchanging heat through the common wall while each keeps its own thermostat — the back zone conditioned, the sunspace riding free between a winter night around 7 °C and a summer afternoon near 49 °C, both inside the reference programs' range.

The results, next to everyone else’s

The charts below show Aladdin's annual heating and cooling loads for all twenty-one conditioned zones, split into lightweight and high-mass rows (Case 960 charts its sunspace-fed back zone), plotted directly alongside the six reference programs from the standard's example results. The shaded strip behind each cluster spans the reference minimum to maximum. Hover over any bar for its exact value.

The headline: Aladdin's annual heating falls inside the reference range on all thirteen conditioned zones of the core suite — lightweight, high-mass, and the two-zone sunspace alike, including the notoriously demanding thermostat-setback cases — and so does every one of their hourly peak-heating loads. The extended variants (increased insulation, zero-deadband thermostats) land inside or within about five percent; their absolute loads are roughly half as large, so the same small biases read as bigger percentages. Annual cooling lands inside the range on the entire lightweight series and on most of the high-mass series, where the remaining cases sit a few percent below the range bottom. The free-float temperature extremes, from a −12 °C winter night to a 68 °C summer greenhouse spike in the unconditioned box, fall inside the published ranges as well. Counting every reported metric across all twenty-seven cases — ninety-eight numbers spanning annual loads, hourly peaks, and free-float extremes — two thirds fall strictly inside the reference ranges and, apart from the tiny night-ventilation cooling load of Case 950, every miss is within about eight percent of the nearest bound (the free-float misses are fractions of a degree) — smaller than the disagreement among the reference programs themselves on many outputs.

And where Aladdin still deviates, we can tell you exactly where and why, because the case structure localizes it: the high-mass buildings run their air conditioners a few percent less than the references — most visibly in the night-ventilation case (950), which pre-cools its concrete slab a bit too effectively and undershoots the smallest cooling number in the suite — and the lightweight peak-cooling hour comes in three to four percent above the range on the south-glazed cases. The two-zone sunspace case follows the same pattern: its back-zone heating and the sunspace’s temperature swing land inside the ranges, while its cooling reads a few percent low — partly the same high-mass cooling signature, and partly because the standard assigns a fifth of the sunspace’s transmitted solar to the massive common wall, whose conduction into the back zone air-coupled zones deliver more weakly. The low-e window case underheats by six percent: Aladdin models that window as a constant air-to-air U-value with the standard’s fixed film coefficients, while the reference programs resolve the pane-gap physics — temperature-dependent gas and radiative conductance, hourly films — and at this window’s small U and small loads, tenths of a watt per square meter are worth several percent of the heating bill. The other extended variants stretch the familiar residuals rather than reveal new ones — the super-insulated buildings simply magnify the lightweight biases in percentage terms — and the free-float extremes sit within a fraction of a degree of the reference minima. All of it is documented rather than papered over. In validation work, knowing precisely where your model bends is nearly as valuable as the places it doesn't.

What the test taught us

The most valuable thing about BESTEST is not the scorecard — it is what the process forces you to find. Chasing the reference ranges through the case structure, month by month and finally hour by hour against the standard's published hourly data, uncovered a series of genuine physics upgrades, each traceable to a source rather than tuned to a benchmark. The diffuse sky model gained the circumsolar brightening that real skies have (an anisotropic, Hay-model sky, which also sharpened every solar panel simulation in Aladdin). The windows gained the true angle-dependent optics of double glazing, taken from the standard's own LBNL WINDOW data. The transmitted beam strikes the floor first and reflects diffusely off it, exactly as the standard's own interior-solar-distribution construction describes. The floor slab became its own thermal node — and then the slab and the walls each split into a surface layer and a core behind the concrete's own conduction resistance, so a sun patch superheats the surface it lands on while the core charges and discharges on its real multi-hour timescale. The room air is now coupled to the building fabric by convection only — the radiative part of the interior films flows between surfaces, as it physically must — and that convective film itself now varies with the air-to-surface temperature difference, following the buoyancy-driven natural-convection correlation the reference programs use, so a room whose surfaces have equilibrated with the air decouples from the fabric the way a real still room does. And the thermostat and ventilation controls gained the linear setback ramp and the scheduled vent fan that the standard specifies, both now available to every Aladdin model, not just these.

That is the real dividend of a method-of-test standard: every deviation you chase down makes the engine better for everyone who will never open a benchmark. The same physics that now reproduces a concrete test box in Denver is what simulates a student's passive-solar house design, a school building with a ground-source heat pump, or a home with a battery and a rooftop array.

The uniqueness of Aladdin

One more contrast is worth stating plainly. Of the six reference programs, none runs in a browser: all are desktop installations, and two — EnergyPlus and CSE — are simulation engines driven by text input files, with graphical front-ends left to third-party tools. TRNSYS, the most established commercial option, costs several thousand dollars per seat. Aladdin is free, runs in a browser tab with nothing to install, treats an interactive 3D model as the primary interface rather than an input file, and ships with a built-in AI assistant — a combination none of the reference programs offers. The point of the benchmark is that this accessibility costs nothing in physics: the same tool a student opens in a classroom stands next to the engines the field trusts.

Try it yourself

Every case in this article can be run in a browser. Open Aladdin, choose Benchmarks → Building Energy from the main menu, pick a BESTEST case, and run the dynamic yearly analysis — the scene notes describe what each case isolates, and the thermostat, ventilation, and thermal-mass settings behind each one are right there in the building's HVAC dialog to inspect and change. Turn the thermal mass up and watch the heavy building's cooling collapse; remove the overhang and watch the solar gain flood back in. The benchmark scenes are pinned to the exact TMY3 weather file the reference programs used (the panel header names it), and the dynamic yearly analysis simulates all 365 days in sequence — so the totals you get reproduce the table above, not merely approximate it. Ordinary scenes are unaffected: they keep fetching current typical-year weather for their own locations. The box was designed to keep simulators honest, but it turns out to be a rather good teacher, too.

← Back to Aladdin