Depth-guided generation
Generate posed RGB views from depth context and target cameras.
atlasChisel generates posed RGB views from optional posed depth context, a
required text prompt, a set of target cameras, and optional posed photographs
of the scene that the generated views stay consistent with. A context frame
may also carry the view's known pixels, a mask, and a depth mask, in which
case the task keeps the known pixels and depth and generates the rest of the
view.
Account availability:
POST /api/v2/tasks:atlasChiselrequires access to this task on your account.
When to use it
Use this task when you have a graybox, blockout, or another geometric layout
expressed as posed depth images and want Marble to render coherent RGB views at
the same cameras. You can also leave contextFrames empty to generate from the
prompt alone.
The request combines:
contextFrames: zero or one posed frame per target camera, in target order, each carrying depth, the view's known image, or both, optionally with a mask and a depth mask.sourceFrames: zero or more posed RGB photographs of the scene, each an image and its camera.targetCameras: 1 to 1440 pinhole cameras for the views to generate. The current model's rollout holds fewer; see the rollout budget below.prompt: a required description of the environment, materials, lighting, and style.modelParameters: optionalseedandnumStepsoverrides.voxelSizeandvoxelGridOrientation: optional controls for simplifying the depth geometry onto a grid of cubes (voxels).
atlasChisel serves one model and does not accept a model selector. The
model keeps each frame's known depth and image, limited to the maskAsset
when one is set, and generates the rest. Start with the documented defaults
and change one sampling or voxel control at a time.
Keep the prompt, seed, and camera set with the operation so a result can be
reproduced.
Control layout and appearance
Use the blockout to set object positions, sizes, and shapes, then render depth
at each target camera and pass it in contextFrames. Use the prompt to name
the intended objects and describe their materials, lighting, and style.
For example, keep a table's blockout unchanged and change the prompt from an oak table in warm daylight to a concrete table in cool evening light. To move the table, move its blockout geometry and render new depth for every target view; changing only the prompt leaves the old position in the depth input.
Keep cameras and modelParameters.seed fixed for comparisons. The same seed
is passed to the default prompt enhancer, so enhancePrompt can stay enabled.
Exact output matches are not guaranteed.
Camera and depth requirements
Every context and target camera must be a 1280 x 720 pinhole camera. Each context depth raster must also be exactly 1280 x 720 and aligned with its camera. The public endpoint does not resize or crop depth maps for you.
When context is present, provide exactly one frame per target camera in the
same order. Each frame carries camera and at least one of depth and
imageAsset:
depth.depthAssetfor a linear-depth EXR, ordepth.logDepthAssetfor an inverted log-depth PNG (see Encode context depth)imageAsset, the view's known pixels at 1280 x 720, as an 8- or 16-bit image- optionally
maskAsset, an 8-bit grayscale PNG that is white where the frame's image and depth are known and black where the model is free to generate; frames with images and no masks are fully known, and frames with masks and no images keep their depth inside the mask only - optionally
depth.maskAsset, an 8-bit grayscale PNG on the depth grid that is white where the frame's depth is known and black where the model is free to generate the geometry
A depth.maskAsset masks the depth apart from the image, and the frame's
maskAsset then marks only the image, so it needs an imageAsset beside it.
Without a depth.maskAsset, the maskAsset marks the depth too. For
example, to keep the full depth of a scene with the known image limited to
one object, send an all-white depth.maskAsset and the object's mask as
maskAsset. Both masks read a pixel as known above half intensity.
RGB images in contextFrames and sourceFrames may use 8-bit or 16-bit
values. If an image has another bit depth, the operation fails with
INVALID_INPUT before model inference. Convert the image to 8-bit or 16-bit
RGB before submitting another request.
An imageAsset is a finished view of the scene, a photo or a textured, lit
render: the model reproduces its pixels and matches the rest of the view to
them. A render of an untextured blockout or clay model is not that. The model
does not reproduce it and instead restyles it from the prompt in ways you do
not control. For a blockout, send its depth alone, or its depth with a
maskAsset that marks only the pixels whose look is final, and describe the
appearance in the prompt and the style reference.
A frame with an imageAsset and no depth keeps its image inside the
maskAsset and lets the model generate the geometry along with the rest of
the view. This is the frame shape atlasMasked takes, so
a view-completion request moves to atlasChisel by sending its frames as
contextFrames, with the same cameras as targetCameras and a prompt that
either describes the finished scene or says what to change. With
enhancePrompt on, an edit instruction is read against the images and
rewritten as a description of the scene after the edit; a scene description
is expanded from the text alone. Leave
voxelSize unset with these frames: there is no depth to quantize, and the
request is rejected. See
Run the same edit on atlasChisel.
Send each of depth, imageAsset, maskAsset, and depth.maskAsset on
every context frame or on none; a request that mixes frames with and without
one is rejected.
Do not send depth.confidenceAsset to atlasChisel; the endpoint rejects it
instead of silently ignoring it. Keep every camera and depth image in one
coordinate convention and scale. See
Cameras and posed images for the pose
convention, and Euler angles for
converting the Chisel editor's rotation fields.
Three.js EXRLoader: If client code decodes or rewrites a linear-depth EXR before upload, account for
EXRLoader's bottom-up CPU row order. The task expects depth aligned to the camera's top-left image grid. See Decode linear-depth EXR rows for a reference accessor.
Encode context depth
Each depth pixel is planar depth: the distance from the camera to the surface measured along the camera's optical axis (view-space z). It is not the length of the ray through the pixel. Use the same distance unit as the camera positions; the service normalizes the scene scale, so any consistent unit works.
If your renderer or raycaster reports distance along the ray, convert each
pixel before encoding. For ray distance d at pixel center (u, v):
Send each frame's depth in one of two encodings:
| Field | File | Pixel value |
|---|---|---|
depth.depthAsset | Single-channel EXR (image/x-exr) | Planar depth in world units |
depth.logDepthAsset with zMin and zMax | 8- or 16-bit grayscale PNG (image/png) | Inverted log depth: white is zMin, darker is farther |
A log-depth PNG value p from 1 to white decodes as follows, where white
is 255 for an 8-bit PNG and 65535 for a 16-bit PNG:
To encode, set zMin and zMax to the frame's nearest and farthest surface
depth, write round(white * (1 - ln(z / zMin) / ln(zMax / zMin))), and clamp
each surface to at least 1 so the farthest one does not collide with the
no-surface value 0. zMin and zMax must be finite with
0 < zMin <= zMax. If a log-depth PNG has another bit depth, the operation
fails with INVALID_INPUT before model inference. Correct the encoding before
submitting another request.
Mark sky and empty space
Mark every pixel with no surface behind it, such as sky, as no surface:
- in a log-depth PNG, write
0; - in an EXR, write
0,NaN, or infinity.
The service reads any depth that is not finite and positive as no surface.
Atlas's own Chisel editor sends depth this way: planar view-space z, with 0
wherever nothing was drawn.
Do not model the sky as a distant sphere or dome. Any finite positive depth is a surface, so the model reads the dome as far geometry, not as sky. The model also sees depth on a log scale normalized over every context frame in the request together, so a dome kilometers away becomes the farthest surface in the scene and squeezes all real geometry into a narrow band of the depth range.
Keep generated views consistent with photographs
When you already have posed photographs of the scene, pass them in
sourceFrames. Each entry carries an imageAsset and its camera, the same
posed RGB frame shape atlasMasked takes, with no maskAsset and no depth.
The model uses these photographs as visual references for the materials,
colors, and lighting of surfaces visible from the target cameras.
- Cameras must be in the same coordinate convention and scale as the target cameras and any context depth, but they need not match a target camera.
- Any resolution works. Each photograph is cover-resized and center-cropped onto 1280 x 720 before it reaches the model, so keep the subject away from the edges of a photograph with a different aspect ratio.
- On the current model, request sizes must satisfy
3 * contextFrames.length + sourceFrames.length + targetCameras.length <= 128. With one context frame per target, 32 targets leave no room for source photographs; 30 targets leave room for 8 photographs. A larger request is accepted, but its operation fails.
Prompt and voxel controls
prompt must be non-empty whether or not context depth is present. Leave
enhancePrompt at true to expand your prompt into a detailed scene
description, or set it to false to use your wording unchanged.
voxelSize sets the side length of each cube used to simplify the depth
geometry. It uses normalized scene units, not your input's distance units:
Marble rescales the scene so the 90th percentile of inverse depth is
approximately 1. Omit it or set it to 0 to keep the original geometry. The
recommended range is 0 to 0.4; values around 0.05 to 0.15 are useful starting
points for blockouts.
With a nonzero voxelSize, voxelGridOrientation: "world" (the default)
aligns the grid to the request's world axes so walls aligned with those axes
stay flat. "gauge" aligns the grid to the average camera pose after the scene
is recentered.
Handle the result
The task returns a long-running operation. On success, frames contains one
generated image and camera per target camera, in request order.
Set returnDepth: true only with 2 to 256 target cameras whose centers are
not all identical:
- A request with one target camera and
returnDepth: trueis rejected with HTTP422. Context depth frames and source photographs do not count toward the two targets, so a one-camera run cannot return depth. - A request with more than 256 target cameras and
returnDepth: trueis rejected with HTTP422. - Targets that all share one center (a pure-rotation rig) are accepted. With no
anchoring photograph, the operation can still report success with a depth
scale of
0.0orNaN, because it has no translation baseline.
Returned depth is reconstructed from the generated targets, not copied from
the context. The depth comes back from the generation call itself, and context
frames that carry an image and no maskAsset join that reconstruction too, up
to 19 of them, with the depth under a depth.maskAsset left out. Source
photographs guide the generation only: they do not join the reconstruction and
get no depth back. With more than 19 such context frames, the generated views
are reconstructed in a separate step afterwards; there a source photograph
joins as an anchor only when its image is 1280 x 720 and its camera's
principal point is centered.
Get the frames in the submission response
A request sent with ?wait=true and enhancePrompt set to false can come
back from its submission already done: true. By default that response names
each output file by assetId and url. Set base64Outputs to true to
receive each file in that response as a base64 data URL instead. The service
then skips storing the files before it answers, and you skip downloading them. A
later read of the operation names the files by assetId and url either way.
Frames answered at submission are JPEG; frames the operation generates later are
PNG. Depth answered at submission is half-float EXR, which reads any depth above
65504 as infinity; depth the operation produces later is float32 EXR. See
Operations.
Example completed output
After the operation reports done: true, its task-specific result is in
operation.response:
When returnDepth is true, each generated frame also carries a depth
buffer. Its depth can contain NaN where the reconstruction found no reliable
geometry; see
Missing values in returned depth.
Use the Operations polling pattern, and open the atlasChisel reference for the live request schema, defaults, and response types.