The Problem
NeuralOS addresses the need to simulate operating system GUIs through neural generative models. The framework predicts screen frames from user inputs (mouse movements, clicks, keyboard events), enabling research into adaptive, generative neural interfaces without requiring direct system interaction. This is particularly valuable for scenarios where real system data collection is impractical, expensive, or raises privacy concerns.
What This Does
This repository implements NeuralOS, a two-component system: an autoencoder that compresses 512×384×3 RGB images into 64×48×16 latent representations, and a main model that combines an RNN (tracking computer state) with a diffusion-based neural renderer generating desktop images from user input sequences. The autoencoder training code lives in autoencoder/main.py with configuration files like configkl4lr4.5e6loadacc1512384mar10keyboardinit16contmar15acc1cont1e6.yaml. The computer model training code is in computer/main.py with configs under computer/configs/. A suite of visualization tools exists in autoencoder/visualizereconstruction.py and computer/ for analyzing latent spaces and reconstruction quality. Data pipeline includes 15+ preprocessdataset.py scripts and sample.py for generating interaction sequences.
How To Use It
Setup: Build the Docker image for data collection:
cd data/datacollection docker build -t syntheticdatagenerator .
Collect training data:
python syntheticscript.py
Process collected videos and actions:
cd data/dataprocessing python main.py
Train the autoencoder (requires ~1 week on 1 H200 GPU):
cd autoencoder/ python main.py --config configkl4lr4.5e6loadacc1512384mar10keyboardinit16contmar15acc1cont1e6.yaml
Process raw images to latent using the trained autoencoder:
cd autoencoder/ python preprocessdataset.py
Train the main NeuralOS model:
cd computer/ python main.py --config configs/fbcomputecanadachallengingandsamplepretrainrnnbalancedlr5e6contbestsamplercovernewdcontfreezernnnewnewdorigunetnospatialonlinex0jointonlineonly7.yaml
Configuration: Modify data paths in the respective config files to point to your data files. Main model training uses 8 GPUs with FSDP and supports staged training via config flags ("pretrain", "freezernn").
Real-World Use
A research team studying human-computer interaction can use NeuralOS to generate synthetic GUI sequences for training downstream policies without collecting real user data. For example, to simulate application launches and mouse interactions across diverse desktop configurations: load a trained autoencoder checkpoint, run sample.py with keyboard/mouse input vectors, and decode latent images back to RGB via the diffusion renderer. The visualizereconstructionkl512384mar10_keyboard.py script demonstrates this workflow end-to-end.
Code Health & Issues
No CI/CD pipeline - .github/ directory or CI configuration absent; no automated build/test gate exists to validate changes across the 200-file codebase. No LICENSE file - root directory lacks licensing text, creating uncertainty about usage rights and redistribution permissions. Configuration sprawl - 70+ YAML config files in autoencoder/ and 200+ in computer/configs/ make it difficult to discern which configs are active, tested, or production-ready. No test files - only 1 test file detected across the entire repository; critical data processing and model training paths lack automated verification. Dependency hygiene - no package.json, pyproject.toml, or requirements.txt visible in the structure; dependency versions and environments inferred only from shell scripts and config references.
The Bottom Line
NeuralOS is a well-structured research codebase for GUI simulation via neural generative models, with clear data pipeline stages and reproducible training configs. However, the absence of a license, CI/CD, and test coverage limits its readiness for production or collaborative use. It is best suited for individual researchers or teams comfortable managing manual build steps and config selection. The autoencoder alone requires a week on premium hardware, and the main model scales to multi-GPU FSDP training, making it resource-intensive.