Research Explained #5 - When Is a Computer Result Really Independent?

Suppose we run a scientific calculation on a computer and get an interesting result.

Then we run it again.

Same answer.

We run it on another computer.

Same answer again.

We rewrite part of the calculation using a different numerical method. Still the same answer.

At some point it is tempting to say:

“We reproduced the result.”

But in science, that phrase can mean something much stronger than simply getting the same answer twice.

This distinction has become increasingly important in the Beyond the Light Barrier project as we have developed simulations, verification procedures, cross-platform testing, and packages intended eventually for outside reproduction.

The central question is surprisingly simple:

How independent does another calculation need to be before it can genuinely increase our confidence in the result?

---

Running It Again Is Important — But It Is Repeatability

The simplest check is to run the same program again with the same inputs.

If it produces the same result, that is useful.

It tells us something about whether the calculation is repeatable under those conditions. It can also expose problems involving randomness, uninitialized state, unstable numerical behavior, or other execution issues.

But it does not provide much protection against a mistake already built into the program.

Imagine that a programmer accidentally enters:

2 + 2 = 5

Running that same incorrect calculation a thousand times will still produce the same incorrect answer.

Repeatability asks:

“Can we get the same result again?”

That is an important question.

It just isn't the only one.

---

What If We Run It on Another Computer?

Beyond the Light Barrier has been deliberately developing the ability to run research software in substantially different computing environments.

Our current environments include:

• Apple Silicon ARM64 running macOS;
• physical x86-64 hardware running Windows; and
• an IBM s390x big-endian Linux environment running through emulation.

Those differences can be extremely useful.

Software sometimes contains assumptions that remain invisible when it is used on only one platform. File paths, text encoding, line endings, processor architecture, byte order, filesystem behavior, dependencies, and numerical implementation details can all expose problems.

If a calculation survives those differences, our confidence in the software implementation may increase.

But there is an important limitation.

If every computer is still executing essentially the same algorithm derived from the same assumptions, they can all reproduce the same conceptual mistake.

Cross-platform verification therefore answers another question:

“Does the implementation behave correctly across the environments we claim to support?”

That is valuable verification.

It is not automatically independent scientific reproduction.

---

What If We Calculate It a Different Way?

Now the test becomes stronger.

Suppose one calculation uses one numerical method and another implementation uses a substantially different method.

If both methods converge on the same value, we have reduced the chance that the result exists only because of a particular algorithm.

We have used this kind of approach within Beyond the Light Barrier.

For example, some of our computational work has been checked using numerically distinct methods rather than relying on a single implementation.

That is stronger than merely pressing “run” twice.

We call this kind of work internal cross-validation.

But the word internal matters.

The implementations may still share:

• the same equations;
• the same interpretation of those equations;
• the same assumptions;
• the same specification;
• the same researcher;
• the same AI-assisted research process; or
• the same expected answer.

Agreement between the implementations is useful evidence that we may have implemented the mathematics correctly.

It still does not necessarily tell us whether both implementations inherited the same mistake from farther upstream.

---

What About Asking Another AI?

This question is especially relevant to Beyond the Light Barrier because artificial intelligence is used extensively as a research tool within the project.

We can ask another AI model, another reasoning process, another agent, or even another conversation to review a calculation.

That can be extremely useful.

A separate review may notice an assumption, algebraic mistake, software defect, missing boundary condition, or logical problem that the original analysis missed.

In fact, adversarial review is something we deliberately use.

But changing the AI does not automatically make the scientific work independent.

Two AI systems may receive the same specification, rely on similar assumptions, use similar training-derived mathematical knowledge, or be shown the same expected result.

Agreement between them can strengthen internal confidence.

Disagreement can be even more useful because it tells us where additional investigation is needed.

But neither agreement nor disagreement between AI systems determines which answer is physically correct.

The mathematics, evidence, assumptions, and eventually nature still have to decide.

---

So What Counts as Independent Reproduction?

For Beyond the Light Barrier, we deliberately set a stronger standard.

An independent reproducer should be able to begin from the equations, assumptions, parameters, and specification rather than simply receiving our source code and running it.

Where practical, the reproducer should also introduce meaningful independence by changing one or more aspects of the calculation, such as:

• the numerical method;
• the programming language;
• the mathematical reduction;
• the numerical library;
• the discretization;
• the implementation; or
• the researcher performing the work.

The purpose is not to make reproduction unnecessarily difficult.

It is to create opportunities for hidden assumptions and implementation mistakes to reveal themselves.

If two calculations share everything except the person pressing the button, they may not be as independent as they appear.

---

Why Blinding Can Matter

There is another subtle problem.

Suppose we tell an independent researcher:

“The answer should be 2.027901181.”

Now imagine their calculation produces 2.01.

They inspect their code, change something, and obtain 2.025.

Closer.

They make another adjustment and obtain 2.0279.

Eventually their calculation agrees beautifully with ours.

But what happened?

Perhaps they genuinely found several errors.

Or perhaps knowledge of the expected answer influenced decisions about algorithms, tolerances, assumptions, or corrections.

This is one reason a blind comparison can be valuable.

The independent reproducer can perform the calculation and commit the result — including numerical uncertainty and relevant verification checks — before seeing the project's reference answer.

Only then are the results compared.

Agreement becomes more informative because the reproducer was not steering toward a known target.

And disagreement is preserved rather than quietly tuned away.

---

What If the Independent Result Disagrees?

This may be the most important part.

A disagreement is not a failed reproduction exercise.

It is a scientific result that needs investigation.

Perhaps our calculation is wrong.

Perhaps theirs is wrong.

Perhaps the specification is ambiguous.

Perhaps two mathematically reasonable interpretations produce different answers.

Perhaps the result is sensitive to a numerical approximation that nobody previously recognized.

Whatever the cause, changing parameters after the fact merely to force agreement would destroy much of the value of independent reproduction.

The disagreement should instead be preserved and investigated.

For Beyond the Light Barrier, this is especially important because the purpose of reproduction is not to confirm what we hope is true.

It is to discover whether the result survives scrutiny outside the process that originally produced it.

---

Different Tests Answer Different Questions

It helps to think of these activities as answering different questions rather than simply forming a ladder where every step means “more correct.”

Repeatability:
Can the same computation produce the result again?

Cross-platform verification:
Does the implementation behave correctly across different supported computing environments?

Internal cross-validation:
Do different methods or implementations within the project reach compatible results?

Adversarial review:
Can another reviewer, AI system, or analysis process find weaknesses that the original process missed?

Independent reproduction:
Can a meaningfully independent implementation reproduce the result from the scientific specification?

Physical experiment:
Does nature actually behave in the way the model predicts?

Those questions are related.

But they are not interchangeable.

Figure 1. Same Result — Different Levels of Confidence. Repeatability, cross-platform verification, internal cross-validation, adversarial review, independent reproduction, and physical experiment answer different scientific questions. Agreement at one stage can strengthen confidence, but does not automatically establish the evidentiary status of another.

Created for the Beyond the Light Barrier research project with OpenAI image-generation assistance, September 2026.

---

Where Beyond the Light Barrier Currently Stands

This distinction has a practical consequence for how we classify our own work.

The Beyond the Light Barrier Research Method uses nine evidence and research categories. For computational work, three of them are particularly relevant here:

• Category 6 — analytic estimates or calculations;
• Category 7 — numerical or simulation results; and
• Category 8 — independently reproduced results.

Some BTLB calculations have undergone substantial internal verification, multiple numerical approaches, software testing, and cross-platform verification. Those activities strengthen the work, but none by itself satisfies the project's requirements for Category 8 independent reproduction.

But as of this writing:

Category 8 independent scientific reproduction: NONE.

That is not an embarrassing blank space in the research record.

It is an accurate description of where the evidence currently stands.

Calling internal verification “independent reproduction” would make the project sound more advanced than it actually is.

We would rather preserve the distinction.

---

And Independent Reproduction Still Isn't Experiment

There is one final boundary that matters even more.

Suppose an outside researcher independently reproduces one of our computational results perfectly.

That would be significant.

It could show that the result does not depend on our particular implementation or internal research process.

But it still would not demonstrate that the modeled phenomenon exists in nature.

A mathematical model can be calculated correctly by ten independent researchers and still describe a physical possibility that nature does not actually realize.

That is why Beyond the Light Barrier separates independent computational reproduction from physical experimental evidence.

Eventually, where scientifically justified, predictions must confront calibrated physical measurements, conventional explanations, controls, uncertainty, and experiment.

The computer can tell us what the model predicts.

Independent reproduction can tell us whether someone else can obtain that prediction without simply inheriting our implementation.

Nature still gets the final answer.

---

Why We Want Someone Else to Check Our Work

It would certainly be encouraging if an independent researcher reproduced one of our results.

But agreement is not the only valuable outcome.

We don't want someone else to reproduce our answer merely because agreement would make us feel better about it.

We want them to independently ask the same mathematical question.

If they get the same answer, our confidence increases.

If they don't, that may be even more valuable.

Because somewhere inside that disagreement may be an assumption, approximation, ambiguity, or mistake that neither side would have discovered alone.

And finding it is exactly what scientific verification is supposed to do.

---

Research Status

This article explains the verification and reproduction framework used by the Beyond the Light Barrier project. It does not report a new physical experiment, independently reproduced scientific result, or evidence for faster-than-light communication or propulsion.

BTLB currently has internally verified analytic and computational results, including Category 7 numerical work, but no project result currently qualifies as Category 8 independent scientific reproduction or Category 9 physical experimental evidence.

Cross-platform verification, alternate internal implementations, software testing, and AI-assisted adversarial review may strengthen confidence or reveal defects, but they do not by themselves change the evidence category of a scientific result.

---

Sources & Further Reading

Beyond the Light Barrier Research Method, Version 1.2.
Defines the project's evidence categories, reproducibility requirements, simulation standards, claim-strength rules, and distinction between computational and physical evidence.

Beyond the Light Barrier Verification Roadmap, Version 1.1.
Defines the project's transition from internal verification toward independent reproduction, including independence-preserving comparison, convergence requirements, and preservation of disagreement.

Beyond the Light Barrier Software Development Standards, Version 1.1.
Defines the software-verification, cross-platform, AI-assisted review, reproducibility, provenance, and scientific-integrity requirements used in BTLB research software.

Beyond the Light Barrier Research State, Version 1.11.
Records the current project evidence state, including the distinction between internally verified Category 7 work and the absence, as of this version, of Category 8 independent scientific reproduction or Category 9 physical experimental evidence.

Research Explained #3 - When Is a Simulation Actually Evidence?

Much of the work in Beyond the Light Barrier happens on computers.

We solve equations, compare mathematical models, run numerical simulations, test software, and sometimes deliberately try to make that software fail.

That raises an important question:

When does a simulation actually become scientific evidence?

The answer depends on what we mean by evidence.

A simulation can provide evidence about what a mathematical model predicts. It can help us discover mistakes, compare competing models, identify interesting parameter ranges, and decide whether a physical experiment might be worth attempting.

But a simulation cannot, by itself, demonstrate that nature actually behaves according to the model being simulated.

A simulation can tell us what a model predicts.

It cannot, by itself, tell us that nature follows the model.

What Is a Simulation?

A simulation is a computational representation of a model.

Usually we begin with assumptions about a physical system and describe those assumptions mathematically.

A computer then evaluates those equations under specified conditions.

For example, suppose we want to predict the motion of a charged particle in an electric field.

We can describe the particle's motion using established equations of physics, give the computer values for quantities such as the particle's mass, charge, electric field strength, and starting conditions, and calculate how the model says the particle should move.

The resulting trajectory is a prediction of the mathematical model.

It is not a measurement of a physical particle.

The Model Comes First

Before we can understand what a simulation tells us, we need another term: model.

A scientific model is a simplified description of some aspect of reality.

Models can include equations, assumptions, approximations, boundary conditions, and definitions of the quantities being studied.

Even extremely successful physical theories are represented by models when we apply them to particular situations.

A simulation therefore does not simply ask a computer:

“What will nature do?”

It asks something much more specific:

“Given these equations, assumptions, parameters, and starting conditions, what does this model predict?”

That difference is fundamental.

A Computer Can Give a Precise Answer to the Wrong Question

Computers are extremely good at performing calculations.

But they do not automatically know whether the assumptions behind those calculations are physically correct.

Imagine that we create a mathematical model containing an incorrect assumption.

We could write excellent software.

The program could run perfectly.

Every calculation could be performed with extraordinary numerical precision.

We could produce beautiful graphs.

And the result could still fail to describe nature because the underlying model was wrong.

Correctly calculating the consequences of an assumption does not establish that the assumption is physically true.

So How Do We Know the Simulation Is Calculating Correctly?

This brings us to verification.

In scientific computing, verification asks whether the equations and algorithms are being implemented and solved correctly.

In simpler language:

Did we calculate what we intended to calculate?

There are many ways to investigate that question.

We can compare the program with problems whose answers are already known.

We can calculate the same result using different numerical methods.

We can test limiting cases where the equations should reduce to something simpler.

We can deliberately supply unusual inputs and see whether the program handles them correctly.

We can also examine whether increasing the numerical resolution causes the result to approach a stable answer.

What Is Convergence?

That last idea is called convergence.

Many scientific calculations cannot solve a continuous physical problem exactly. Instead, the computer divides the problem into many smaller numerical steps.

Suppose we calculate a trajectory using 10 steps.

Then we calculate it using 100 steps.

Then 1,000.

Then 10,000.

If the numerical method is behaving properly, we may see the calculated answer approach a stable value as the steps become smaller.

That behavior gives us evidence that the numerical approximation is approaching the solution of the equations we intended to solve.

But notice what convergence does not establish.

It does not establish that those equations describe a new physical phenomenon.

It strengthens confidence in the calculation.

Verification and Validation Are Different Questions

Another important term is validation.

Verification and validation sound similar, but they ask different questions.

Verification:
Did we solve the equations correctly?

Validation:
Does the model adequately describe the physical system we are trying to understand?

Validation requires comparison with physical observations, measurements, or experiments within the conditions where the model is being evaluated.

This distinction becomes especially important when investigating speculative physics.

A mathematical model might be internally consistent.

A computer implementation might correctly solve its equations.

Multiple numerical methods might agree.

And none of those facts alone establishes that the proposed phenomenon exists in nature.

What About Repeating the Calculation?

Repetition is also important, but not every kind of repetition tells us the same thing.

Suppose we run the same program twice and obtain the same answer.

That can tell us something useful about the program's repeatability.

But both runs could contain exactly the same mistake.

Now suppose we implement the same equations using two different numerical methods.

Agreement between them gives us stronger evidence that a particular numerical method is not solely responsible for the result.

But both implementations may still share the same assumptions or even the same misunderstanding of the equations.

That is why we also care about independent reproduction.

What Does Independent Reproduction Mean?

Independent reproduction means that another researcher or team attempts to reproduce a result without merely rerunning the original calculation exactly as it was provided.

Ideally, they independently interpret the equations, implement the method, perform the calculation, and compare their result with the original.

If they obtain the same result, confidence in the computational finding increases.

If they obtain a different result, that is scientifically useful too.

The disagreement tells us that something needs to be investigated.

Reproduction is not valuable because everyone is supposed to agree.

It is valuable because disagreement can expose something we missed.

Where Does BTLB-SIM Fit Into This?

BTLB-SIM is the computational framework being developed for Beyond the Light Barrier.

Its purpose is not to make speculative physics look convincing.

Its purpose is to help us ask computational questions in a way that preserves the information needed to understand, verify, challenge, and potentially reproduce the results.

That includes things such as:

• the equations being solved,
• the assumptions being made,
• units and reference frames,
• parameters and starting conditions,
• the numerical method,
• the validity limits of the model,
• uncertainty information,
• software and model versions,
• provenance showing how a result was produced, and
• verification records.

Provenance is simply the record of where a scientific result came from—what model, software, inputs, assumptions, and processes produced it.

Without that information, a number produced by a computer can become very difficult to evaluate scientifically.

Why Are We Trying to Break Our Own Simulator?

This is also why some BTLB-SIM development has involved deliberately trying to make the system fail.

We want to know what happens if a calculation is interrupted.

What happens if an artifact is corrupted?

What happens if part of a scientific record is incomplete?

What happens when unusual or hostile inputs are supplied?

What happens when different processes try to change something at the same time?

These tests do not tell us whether wormholes or warp drives exist.

They test whether the computational infrastructure can preserve a trustworthy record when things do not go according to plan.

Reliable scientific software does not make a speculative physical idea true.

It helps us determine more reliably what the idea actually predicts.

Then What Is a Computational Experiment?

We sometimes use the term computational experiment.

That does not mean a physical experiment happened inside the computer.

A computational experiment systematically varies parameters, conditions, or models and observes the resulting numerical behavior.

In that sense, we can experiment with the mathematical model.

But the object being tested is still the model and its computational consequences.

No physical wormhole appears because a wormhole simulation runs.

No spacecraft travels faster than light because a warp geometry is calculated.

No particle accelerator has been operated because we simulated a particle trajectory.

Computational experiments and physical experiments answer different questions.

What Counts as Physical Experimental Evidence?

Eventually, if a model makes a measurable prediction and the experiment is scientifically and practically justified, we can test that prediction in the physical world.

A physical experiment uses actual apparatus, instruments, controls, calibration, and measurements to observe what happens in nature.

Even then, one surprising measurement is not automatically proof of an extraordinary new phenomenon.

We have to consider measurement uncertainty, instrument behavior, conventional explanations, environmental effects, experimental controls, and whether the result can be reproduced.

But this is the stage where the question fundamentally changes.

We are no longer asking only:

“What does our model predict?”

We can begin asking:

“What did nature actually do?”

A Simulation Can Still Be Important Evidence

None of this means simulation results are unimportant.

Quite the opposite.

Computational results can provide strong evidence about mathematical and numerical questions.

They can show that a proposed model behaves differently than expected.

They can expose instabilities.

They can reveal enormous energy requirements.

They can identify parameter ranges where an effect becomes too small to measure.

They can eliminate proposed experiments before time and money are spent building them.

And sometimes the most valuable result of a simulation is:

“This probably isn't going to work.”

That is still scientific progress because it constrains where we should look next.

The Evidence Should Match the Claim

This leads to the principle we try to preserve throughout Beyond the Light Barrier:

The evidence should match the claim.

If we have performed an analytic calculation, we should call it an analytic calculation.

If we have performed a numerical simulation, we should call it a numerical simulation.

If someone independently reproduces it, we should record that separately.

If we eventually perform a physical experiment, that should be identified as physical experimental evidence.

Moving from one category to another requires additional evidence.

Excitement about the result does not move it up the ladder.

So When Is a Simulation Evidence?

The answer is:

Once the simulation has been sufficiently verified for the claim being made—but only for the claims it actually supports.

A properly constructed simulation can provide evidence about the consequences of a mathematical model.

Verification can increase our confidence that the calculation was performed correctly.

Independent reproduction can increase our confidence that the result is not merely an artifact of one implementation.

Comparison with physical measurements can begin testing whether the model describes nature.

Each stage answers a different question.

Keeping those questions separate may make extraordinary research sound less dramatic.

But it makes the conclusions much more meaningful.

The computer can tell us what our model says.

Nature gets the final answer.

---

Research Status

Current status: This article explains how numerical and simulation results are evaluated within the Beyond the Light Barrier Research Method. It does not report a new physical discovery.

• Evidence Category 7 — Numerical/simulation results: Simulations can provide computational evidence about the behavior of a specified model once they have been sufficiently verified for the claim being made. Such results remain conditional on the model assumptions, equations, parameters, boundary conditions, and numerical implementation.
• Evidence Category 8 — Independently reproduced results: Independent reproduction can strengthen confidence that a computational result is not specific to one implementation, codebase, numerical method, or investigator. This article does not itself report a new independently reproduced result.
• Evidence Category 9 — Experimental results: Comparison with physical measurements or experiments is required to test whether a model adequately describes nature within the conditions being evaluated. This article does not report a new physical experimental result.

A verified or independently reproduced simulation should therefore be interpreted as evidence about the specified computational model at the level justified by its verification and reproduction status—not as evidence that a speculative physical phenomenon necessarily exists in nature.

---

Sources & Further Reading

ASME. Verification, Validation, and Uncertainty Quantification Terminology in Computational Modeling and Simulation (VVUQ 1-2022). American Society of Mechanical Engineers.

Oberkampf, William L., and Christopher J. Roy. Verification and Validation in Scientific Computing. Cambridge University Press, 2010. DOI: 10.1017/CBO9780511760396.

Roache, Patrick J. “Quantification of Uncertainty in Computational Fluid Dynamics.” Annual Review of Fluid Mechanics, Vol. 29, 1997, pp. 123–160. DOI: 10.1146/annurev.fluid.29.1.123.

Beyond the Light Barrier. Beyond the Light Barrier Research Method, Version 1.1. Project research-method framework for evidence classification, reproducibility, simulation, and experimental interpretation.