Scientific computing has a familiar software-engineering problem:
It works on my computer.
For ordinary software development, that phrase is usually a warning that something about the program depends on the particular machine or environment in which it was developed.
For scientific software, the problem can be more serious.
If a computational result depends unexpectedly on the operating system, filesystem behavior, text representation, or some other platform-specific assumption, then reproducing the calculation on another computer may not mean what we think it means.
BTLB-SIM recently encountered exactly this kind of problem.
What initially looked like a software-portability issue became a useful test of several principles we have been trying to build into the project: preservation of failures, provenance, versioning, verification, and careful separation between software reproducibility and scientific evidence.
Why Another Operating System Matters
Much of the early BTLB-SIM development has taken place on macOS.
That is perfectly adequate for developing software, but it creates an obvious question:
What happens when the same computational experiment is moved to a different operating system?
Windows was therefore introduced as another verification environment.
The objective was not to prove any new physics.
It was to determine whether important portions of the computational framework behaved consistently when moved away from the environment in which they were originally developed.
That distinction matters.
Running the same implementation on another operating system can strengthen confidence in the portability and reproducibility of the software.
It does not constitute an independent derivation of the physics, an independently developed scientific implementation, or a physical experiment.
The First Portability Problem
One of the first issues involved filesystem behavior.
Some durability and synchronization behavior had been implemented using assumptions appropriate to POSIX-style systems such as macOS.
Those assumptions could not simply be carried over to Windows.
This is a good example of why portability testing matters.
The scientific model may be unchanged.
The equations may be unchanged.
The experiment definition may be unchanged.
Yet implementation details beneath them can still behave differently on another operating system.
The solution was not to pretend the difference did not exist.
The architecture had to distinguish the scientific requirement from the platform-specific mechanism used to satisfy it.
Then We Found a More Subtle Problem
Another issue involved something that normally seems almost trivial: line endings.
Text-file line-ending conventions can differ across operating systems and tool configurations.
A line ending commonly represented by a single line-feed byte on one system may be represented by a carriage-return and line-feed pair on another.
To a person reading the file, the text may appear identical.
To a cryptographic hash operating on the raw bytes, however, those files are different.
That matters when hashes are being used to identify frozen scientific artifacts.
If the bytes change, the identity check changes—even when the visible text appears unchanged.
This transformed what might otherwise have been dismissed as a Windows text formatting problem into a provenance and reproducibility problem.
If we say that two computational runs used the same frozen specification, we need a sufficiently precise definition of what “the same” means.
The First Windows Run Failed
The initial Windows-targeted verification did not produce a clean result.
1 test passed. 8 tests failed.
That result is important enough to preserve.
The failures were not evidence that the underlying physics was wrong.
They were also not evidence that Windows was unsuitable for scientific computing.
They exposed a problem in how the historical implementation interacted with Windows-specific behavior.
In particular, the tests were invoking a historical V1 implementation that contained the preserved Windows defect.
Once the problem was understood, it would have been easy to fix the old code, rerun the tests, and allow the failed attempt to disappear from the story.
We deliberately did not do that.
The failed run is part of the development record because it tells us something important about the assumptions that existed at that point in the software.
A failure that teaches us something should not disappear merely because we later learn how to fix it.
Correcting the Software Without Rewriting Its History
The correction raised another question:
What should happen to the implementation that contained the defect?
For BTLB-SIM, the answer was not simply to overwrite it and behave as though the earlier implementation had never existed.
The historical V1 behavior was preserved.
A distinct V2 successor corrected the raw-byte specification-verification behavior.
This distinction matters because scientific provenance includes more than the latest version of the software.
Provenance—the traceable record of the models, assumptions, data, parameters, software versions, and methods that produced a result—also requires us to know which implementation existed when an earlier result was produced.
Silently changing historical software can make that reconstruction much more difficult.
Preserving the old implementation while correcting the successor allows both statements to remain true:
This is what the historical implementation did.
This is how the successor implementation corrected it.
Testing the Successor
After separating portable successor verification from direct comparisons against the historical implementation, the recorded test results became:
These results belong to the specific V2 successor implementation and cross-platform verification scope described here. They should not be treated as verification of every BTLB-SIM version, module, operating system, or future implementation. The associated development record should preserve the implementation version or source-code commit, test configuration, execution environment, and verification results needed to identify and reproduce this verification state.
macOS: 16 passes
Windows: 9 passes, 7 intentional skips, 0 failures
The seven Windows skips are important.
They should not be described as successful tests.
They were intentional exclusions involving historical V1-to-V2 execution comparisons that were not appropriate to count as portable Windows passes under the scoped verification policy.
Recording a skip as a skip is part of keeping the verification record honest.
Six preserved scientific regression cases were also compared using exact Python floating-point equality under this particular verification scope.
That does not establish universal numerical identity across every future platform, compiler, solver, or computational environment.
It records what was actually tested under the defined conditions.
What Did We Actually Verify?
This is where evidence classification becomes especially important.
Successful execution on macOS and Windows provides stronger evidence that the tested portions of the implementation are portable across those two environments.
It also helps reveal hidden operating-system assumptions that a single-platform development process might never expose.
But it does not promote the scientific evidence category of the underlying physics.
Cross-platform agreement is internal software verification.
It is not independent scientific reproduction.
It is not a physical experiment.
The distinction is similar to the one discussed in the previous journal entries about running someone else's software.
If another machine executes the same verified computational artifact under defined conditions and obtains the same output, we have learned something valuable about the reproducibility and portability of that implementation.
We have not automatically obtained an independent derivation, an independent implementation, or independent confirmation of the physical model.
Portability Became Part of the Standards
These problems helped clarify that cross-platform behavior could not remain an informal software-development concern.
It needed to become part of the project's documented development standards.
The resulting BTLB Software Development Standards v1.0 formalize requirements involving cross-platform behavior, raw-byte integrity, canonical representation, reproducibility, provenance, structured outcomes, preservation, versioning, verification, and release records.
One particularly useful distinction is that software status should not be collapsed into a single word such as “complete.”
A component may instead need to be distinguished as:
designed → implemented → tested → platform-verified → independently reproduced
Those are different milestones.
Reaching one does not silently imply that the others have also been reached.
There Is Still an Open Problem
The cross-platform work is not finished.
Another issue, tracked as CP-003, concerns operating-system path separators leaking into persistent provenance.
Windows and POSIX-style systems normally represent filesystem paths differently.
If those native representations are written directly into persistent scientific records, logically equivalent provenance may differ merely because the calculation was executed on another operating system.
The intended direction is to use a canonical serialized representation, including consistent forward-slash path notation where appropriate.
But CP-003 remains open in the current record.
We therefore should not claim that BTLB-SIM has completed cross-platform portability verification.
Some important portability problems have been identified and corrected. Others remain under investigation.
Why This Matters to the Physics
None of these issues changes Einstein's equations.
A Windows line ending does not alter general relativity.
A filesystem path separator does not create or destroy a gravitational effect.
But software is increasingly part of the chain through which we calculate, test, preserve, and communicate scientific results.
If that chain cannot tell us exactly what artifact was executed, what version produced a result, what assumptions accompanied it, or whether moving to another platform silently changed something important, then the scientific record becomes weaker.
That is why apparently mundane engineering details can become research-integrity issues.
Where the Project Stands
The current cross-platform work has demonstrated that important portions of BTLB-SIM can be verified across macOS and Windows under the tested conditions.
It has also exposed assumptions that prevented other portions from being portable without correction.
Historical failures and implementations have been preserved rather than silently rewritten.
Successor behavior has been tested separately from historical behavior.
Additional portability work remains open, including CP-003.
No new physical phenomenon is reported by this work.
No wormhole, anomalous gravitational effect, warp geometry, faster-than-light communication, or faster-than-light propulsion has been experimentally demonstrated.
This is a software-verification and research-integrity milestone.
Beyond “Works on My Computer”
The lesson from this stage of BTLB-SIM development is not that scientific software must produce identical behavior everywhere without qualification.
The lesson is that we need to know when it does not.
We need to understand why.
We need to preserve the failures that reveal hidden assumptions.
We need to distinguish historical behavior from corrected behavior.
And we need to state precisely what successful cross-platform verification does—and does not—demonstrate.
“Works on my computer” is the beginning of a verification question, not the end of one.
Sources & Further Reading
Beyond the Light Barrier — Research Method v1.0.
Project methodology governing evidence classification, assumptions,
uncertainty, reproducibility, preservation of negative and failed results,
corrections, and scientific claims.
Beyond the Light Barrier — BTLB Software Development Standards v1.0.
Project standards governing cross-platform behavior, raw-byte integrity,
canonical representation, provenance, versioning, verification, preservation,
and release records for BTLB scientific software.
Beyond the Light Barrier — BTLB-SIM Framework Implementation State
v0.9-dev.
Development-state record for the BTLB-SIM framework, including implemented
capabilities, verification status, and remaining development work.
Research Status
BTLB-SIM: Active development. This entry documents a cross-platform software-verification and research-integrity milestone rather than a new physical result.
Cross-platform verification: Partial. The specific successor implementation and verification scope described in this entry were exercised on macOS and Windows. Additional portability work remains open, including CP-003.
Historical result: The initial Windows-targeted verification produced 1 pass and 8 failures. That result and the historical V1 behavior are preserved as part of the development record.
Successor verification: Under the scope described in this entry, the recorded results were 16 passes on macOS and 9 passes, 7 intentional skips, and 0 failures on Windows. These results apply only to the tested implementation and conditions and should not be generalized to all BTLB-SIM versions or platforms.
Independent reproduction: No new Category 8 result is established by this work. Cross-platform execution of the same implementation is internal software verification, not independent scientific reproduction.
Physical experiment: No new Category 9 result is established by this work.
Evidence status: Nothing reported here constitutes physical evidence for a wormhole, anomalous gravitational effect, warp geometry, faster-than-light communication, or faster-than-light propulsion.
No comments:
Post a Comment