Fitting Neural Textures and PBR Material Maps with ES (No Backprop)

A small, self-contained C++ experiment: an RGB image (or up to 4 same-size RGB textures of 1 material) is encoded arsenic a shared low-resolution latent texture positive a mini MLP decoder, and some are trained entirely pinch Evolution Strategies — nary backprop, nary autodiff, nary training framework. (An optional late-training polish, --mlp-fd, switches the decoder to numerical finite differences; still nary backprop.) Dependencies are stb_image, stb_image_write, and OpenMP.

Write-up: Fitting a neural texture decoder pinch ES

Z is the latent texture, phi a mini positional encoding. At decode clip each pixel bilinearly samples Z astatine its UV, appends phi, and runs the MLP. An optional second, coarser latent level (--latent2) is sampled astatine the aforesaid UV and its channels are concatenated onto the first level's.

Results

512×512 harvest of kodim23, 3000 iterations, latent quantized to 8 bits aft training, MLP weights counted arsenic fp16:

These runs utilized the original positional encoding uv,fourier:1 (--nfreq 1), which is why the information bid beneath passes it. The existent default is uv only, which scored 0.26–0.32 dB higher wherever some were tally (see METHOD.md); the results quoted later successful this record usage that default unless stated otherwise.

The 128×128×8 tally uses a 14 → 24 → 24 → 3 MLP (1035 weights, leaky ReLU, sigmoid output) and trains successful astir 150 s connected a 32-thread CPU. Quantizing the latent to 8 bits costs 0.04 dB.

Output of that tally (out_128c8/): target crop, reconstruction aft 3000 iterations, and the 8 latent channels broadside by side.

out_128c8/model.bin is the trained model; measure it pinch ntc kodim23.png --load out_128c8/model.bin --latent 128 128 8 --nfreq 1 --iters 0.

A 4-layer material

The PavingStones070 worldly (normal, roughness, albedo, AO; spot Test images) trained jointly from 1 shared 128×128×4 + 64×64×4 latent and 1 10 → 36 → 36 → 12 MLP (2172 weights), 3000 iterations, learning complaint annealed complete the 2nd half, per-weight finite differences for the decoder complete the past quarter. 8-bit latent: 2.64 bpp total, 0.66 bpp per texture. Left is the target, correct the reconstruction from the quantized latent (out_m1234/). The training bid was

out_m1234/model.bin is the trained material; measure it pinch ntc m1.png m2.png m3.png m4.png --load out_m1234/model.bin --latent 128 128 4 --latent2 64 64 4 --mlp 36,36 --iters 0.

How the ES training works

Building

CMake generating a Visual Studio solution (MSVC), aliases immoderate C++17 compiler pinch OpenMP connected Linux/WSL. Tested pinch MSVC 2022 and MSVC 2026 connected Windows, and gcc 13 nether WSL2. Note that std::normal_distribution differs betwixt modular libraries, truthful the aforesaid seed gives somewhat different results connected each platform.

For Visual Studio 2026 usage -G "Visual Studio 18 2026", which needs CMake 4.2 aliases newer (the CMake bundled pinch VS 2026 works).

Running

Several positional images shape a material: they must person the aforesaid size aft cropping, stock the latent and the MLP (3 outputs per texture), and tin beryllium weighted successful the nonaccomplishment pinch --weights (relative; a weight of 0 drops that texture from training). Per-texture PSNRs are printed alongside the wide one, output files summation a _tK suffix, and the bitrate is reported some per worldly pixel and per texture.

Run pinch nary arguments, ntc trains connected the checked-in kodim23.png utilizing the default 64×64×4 latent. Images are located comparative to the executable, truthful this useful from the build directory arsenic good arsenic the repo root. A named image that cannot beryllium recovered is an error; the synthetic fallback only applies erstwhile nary image is named.

Progress is printed to stdout (MSE, PSNR, quantized PSNR and bitrate, latent stats, throughput). Reconstructions, a latent visualization per level, and model.bin are written to the output directory periodically; side_by_side.png (target | 8-bit-latent decode) is written astatine the end.

Useful options (ntc --help lists them all):

Images larger than 512×512 are center-cropped by default (--crop); each textures of a worldly are cropped identically and must lucifer afterwards.

Test images

Prior creation disclosure

The blog station supra and the single-texture results (the original repository, richgel999/neural_texture_es) were published connected September 3, 2026. This repository, including the two-level latent, materials, and the items marked "added September 4", was published connected September 4, 2026. The pursuing are disclosed present arsenic nationalist anterior art.

Neural texture representations utilizing learned latent grids pinch mini neural decoders, and Evolution Strategies / simultaneous-perturbation methods for derivative-free optimization, are established ideas. The technically unique portion explored present is their operation pinch the decoder's known spatial dependency structure: each latent values are perturbed simultaneously, antithetic full-image evaluations nutrient per-pixel nonaccomplishment differences, and each pixel's nonaccomplishment quality is attributed only to the latent texels really publication by that pixel's filtering footprint. This yields simultaneous, support-restricted ES estimates for each latent texel while discarding nonaccomplishment variety from pixels a fixed texel cannot affect. Estimates for neighboring texels still stock pixels and the aforesaid perturbation draw, truthful they are correlated alternatively than independent.

For a fixed latent value, the omitted per-pixel nonaccomplishment position do not dangle connected that value's perturbation, truthful their products pinch it person zero anticipation successful the mean Gaussian ES estimator. Footprint attribution truthful removes them without bias, arsenic a variance-reduction system that follows straight from the decoder's dependency graph.

Implemented successful this repository:

• A low-resolution latent texture positive a mini MLP decoder, pinch both the latent and the decoder optimized wholly by antithetic Evolution Strategies, without backpropagation aliases analytic derivatives (an optional finite-difference polish for the decoder is numerical, not autodiff).
• Support-restricted footprint attribution for latent ES: each latent values are perturbed simultaneously and the afloat image is decoded for +ε and −ε. Each pixel's nonaccomplishment quality is attributed only to the latent texels successful that pixel's bilinear sampling footprint, truthful 1 antithetic decode brace produces simultaneous section ES estimates crossed the full latent while excluding nonaccomplishment position that cannot dangle connected each texel.
• Separate ES schedules matched to parameter support: minibatched, many-pair ES for the globally acting decoder weights, and full-image, few-pair footprint-attributed ES for the spatially section latent, interleaved each iteration, pinch the estimates fed done Adam.
• Late-training decoder phases (added September 4): from a chosen fraction of the run, the decoder measurement tin move to (a) antithetic ES evaluated connected the afloat image, (b) per-weight cardinal finite differences connected a shared minibatch, a numerical gradient pinch nary autodiff, pinch the decoder's Adam authorities reset astatine the move because the ES phase's second-moment estimate different throttles it, aliases (c) nary decoder updates astatine each (latent-only phase). Measured connected mario pinch the 128×128×4 + 64×64×4 configuration, a 36,36 decoder, 64 ES pairs, and the learning complaint annealed complete the 2nd half: finite differences complete the past 4th gained 0.27 dB astatine 3000 iterations and 0.42 dB astatine 6000 (31.38 → 31.80 dB), matching a 12000 loop tally successful half the iterations; the Adam reset much than doubled the effect; the full-image and stiff phases changed thing measurable.
• Learning-rate annealing nether ES (added September 4): decaying some learning rates linearly complete the 2nd half of a run. Because ES gradient sound is re-injected each step, a fixed-rate Adam tally settles astatine a jitter floor; annealing removed astir of a visible texel-aligned artifact and gained 0.85 dB astatine 6000 iterations connected mario pinch a azygous 128×128×8 latent and the ldct:2 positional input (31.36 → 32.21 dB) and 0.56 dB connected the two-level configuration supra (30.82 → 31.38 dB).
• Alternating-level perturbation (added September 4, --lat-alt): pinch 2 latent levels, each antithetic brace perturbs only 1 level, rotating crossed steps, pinch each level's gradient scaled by its ain brace count. This removes cross-level crosstalk successful the footprint attribution exactly; measured pinch a 64×64 2nd level it mislaid 0.17 dB, because halving each level's brace count costs much than the mini crosstalk it removed.
• Post-training scalar quantization of the latent pinch per-channel scale, and reported bitrate astatine arbitrary latent spot depth.
• Pluggable positional encodings for the decoder, including cell-periodic cosine features of the bilinear compartment offset (ldct:N), recovered to amended value astatine good latent resolution.
• Configurable decoder depth, width, and activation; saved models grounds the size of each latent level, MLP layout, activation, positional spec, and texture count (not the output mapping aliases the nonaccomplishment weights; spot METHOD.md §7).
• Two-level latent pyramid (added September 4, --latent2): a 2nd latent texture sampled astatine the aforesaid UV and concatenated onto the first, trained pinch the aforesaid footprint attribution applied erstwhile per level. Measured astatine 3000 iterations, 8-bit latent: kodim23 64×64×4 + 16×16×4 gives 27.47 dB astatine 0.59 bpp versus 27.16 dB astatine 0.55 bpp for 64×64×4 alone; mario 128×128×4 + 32×32×4 gives 29.08 dB astatine 2.18 bpp versus 28.82 dB astatine 2.05 bpp.
• Materials trained by ES (added September 4): up to 4 same-size RGB textures trained jointly from 1 shared latent and 1 MLP pinch 3 outputs per texture, pinch per-texture nonaccomplishment weights and per-texture reporting. Compressing a material's textures jointly from a shared latent is established believe successful neural texture compression; what is disclosed present is doing it wholly derivative-free: the per-pixel nonaccomplishment sums complete each texture's channels earlier footprint attribution, truthful 1 antithetic decode brace of the full worldly yields the support-restricted ES estimate for each latent texel pinch respect to each textures astatine once, and the decoder is trained by ES aliases per-weight finite differences, pinch nary backpropagation done immoderate texture. The weights participate the per-pixel nonaccomplishment earlier attribution, truthful a zero weight removes that texture's power connected the latent gradient exactly. Results astatine 3000 annealed iterations pinch per-weight finite differences complete the past quarter, 8-bit latent: the 4-layer PavingStones070 worldly (normal, roughness, albedo, AO) sharing 128×128×4 + 64×64×4 and a 36,36 decoder reaches 23.2 / 31.5 / 23.2 / 29.6 dB astatine 0.66 bpp per texture (2.64 bpp total); the first 2 layers unsocial scope 25.0 / 32.5 dB astatine 1.31 bpp per texture. Two unrelated photographs (kodim23 + mario) sharing 128×128×8 + 64×64×4 (annealed, nary finite-difference phase) onshore astatine 30.4 and 29.6 dB astatine 2.3 bpp per texture, astir what each gets unsocial astatine a akin per-texture bitrate, arsenic expected erstwhile location is thing to share.

Described, not yet implemented:

• Quantization-aware training nether ES: quantize (or block-compress) the latent wrong the decode utilized for each ES evaluation. Because ES only observes nonaccomplishment values, immoderate non-differentiable quantizer aliases codec tin beryllium successful the loop pinch nary straight-through estimator aliases differentiable surrogate.
• Latents stored successful modular GPU texture formats, wrong the training loop. The latent texture is yet a GPU texture, truthful it tin beryllium held successful immoderate format the hardware samples natively: uncompressed fixed-point formats (A8R8G8B8, R8, RG8, 4-bit and 5:6:5 packings, RGBA16), aliases block-compressed formats (BC1–BC7, BC6H for signed aliases HDR latents, ASTC LDR and HDR astatine immoderate artifact size). Because ES only observes nonaccomplishment values, the format's encode–decode information travel tin beryllium wrong each ES information pinch nary straight-through estimator aliases differentiable surrogate: the trainer sees the latent precisely arsenic the GPU will. For artifact formats, nonaccomplishment attribution is per artifact alternatively than per texel, since 1 endpoint alteration moves each texel successful the block; for per-texel formats the attribution is unchanged.
• Search straight successful the encoded representation. Rather than training a float latent and encoding it, make the encoded texture itself the parameter vector and perturb its stored fields directly: quantized texel values for fixed-point formats, aliases artifact endpoints, partition and mode selectors, and per-texel indices for block-compressed formats, utilizing ES pinch discrete perturbations aliases stochastic coordinate descent pinch accept/reject, precisely arsenic a accepted texture encoder searches. There is past nary encoder wrong the loop astatine all, only the format's decoder, the bitrate is fixed by the format by construction, and the trainer and the texture compressor are the aforesaid programme pinch the MLP wrong its distortion metric.
• Non-overlapping perturbation phases: perturb only texels aliases blocks connected 1 shape of a 2×2 grid per information truthful that, nether bilinear sampling, nary 2 perturbed footprints overlap and neighbour crosstalk vanishes; rhythm the shape to screen each parameters. (Level alternation supra is the cross-level analogue and is implemented; the within-level type is not.)
• Latent initialization from the image itself arsenic a lukewarm commencement for ES training: box-downsample the target (all textures of a material) to each latent level's resolution, past task each texel's stacked transmission vector (3T values, aliases a mini section spot of them) onto the apical C main components of those vectors (PCA), truthful the first latent is the champion C- transmission linear summary of the section image contented alternatively of noise. The decoder past starts by learning the inverse projection, which is adjacent to linear. Also decoder initialization from a antecedently trained model.
• Materials pinch non-RGB transmission counts (single-channel roughness aliases AO, two-channel normals) and normal-map-aware losses.
• Training a worldly done a BRDF (rendering loss): put the shading exemplary wrong the ES evaluation, latent → decoded normal, albedo, roughness and AO → BRDF nether 1 aliases much lights and views → rendered image → nonaccomplishment against the aforesaid rendering of the original maps. Today each representation is fitted pinch its ain pixel MSE and hand-set weights; a rendering nonaccomplishment alternatively weights each representation by really overmuch it changes the shaded result, which is what a crippled really sees, and it requires nary derivative of the BRDF, the reside mapping, aliases thing other successful the pipeline. With a per-pixel shading exemplary (no shadows aliases screen-space effects) the nonaccomplishment stays a sum complete pixels, truthful footprint attribution applies unchanged; effects that publication neighboring pixels enlarge the footprint and are handled the aforesaid measurement the coarse latent level is. Several lights aliases views per information conscionable sum much per-pixel terms.
• SPSA (Simultaneous Perturbation Stochastic Approximation) and Rademacher ES successful spot of Gaussian ES: Spall's SPSA perturbs each parameter by a random ±1 (Rademacher) measurement of size c, evaluates the 2 sides L± = L(θ ± cΔ), and estimates each constituent arsenic ĝ_j = (L₊ − L₋) / (2c·Δ_j). Because Δ_j = ±1, 1/Δ_j = Δ_j, truthful this is (L₊ − L₋)·Δ_j / (2c): precisely the antithetic ES estimator utilized present pinch a ±1 guidance alternatively of a Gaussian one, classically pinch a azygous brace per update. Two evaluations estimate the full gradient sloppy of parameter count, and the perturbation packs arsenic 1 spot per parameter. It plugs into the footprint attribution unchanged, since the per-texel nonaccomplishment differences do not dangle connected the perturbation distribution, and the section in installments duty should make it acold little noisy than world SPSA. The planned research is antithetic Rademacher perturbations positive the footprint attribution, for the latent and for the decoder, benchmarked against the Gaussian ES utilized now.
• Learned interpolation kernels expressed arsenic a fewer world parameters alternatively than arsenic decoder inputs.

Status

This is simply a deliberately elemental investigation artifact for learning and experimentation, not a codec. Nothing is tuned. Obvious adjacent steps: quantization-aware training, block-compressed (BC/ASTC) latents wrong the training loop, non-RGB transmission counts for materials (single-channel roughness/AO, two-channel normals) pinch normal-map-aware losses, and replacement losses.