Prompting for Higher Success Rates¶

The following recommendations are based on our experience and experimentation with multiple models and techniques.

Basic Prompt Structure¶

Your prompt is injected into a system prompt that already handles image structure. Object retention is already handled, so write only the change you want.

Choosing Input Images¶

The best images for Neural Rendering are those that have the subjects in clear view, like this one:

A good input image with both vessels in clear view

Prompts applied to it produce consistent generations across the board:

Add aurora borealis in the sky as soft, flowing bands of light in green and
purple tones. The aurora should appear only in the sky and remain diffuse
and atmospheric, without forming sharp edges or distinct shapes.

Four consistent aurora borealis generations from the good input image

Add snow falling from cloudy skies

Four consistent snowfall generations from the good input image

Create a port backdrop on the right side of the image

Four consistent port backdrop generations from the good input image

If the subject is heavily occluded or hard to see, NR may end up warping some of the subjects or perspectives in the outputs. The same prompts applied to this dark frame produce far less consistent results:

A bad input image where the vessels are barely visible

Add aurora borealis in the sky as soft, flowing bands of light in green and
purple tones. The aurora should appear only in the sky and remain diffuse
and atmospheric, without forming sharp edges or distinct shapes.

Four inconsistent aurora borealis generations from the bad input image, with the vessels displaced

Add snow falling from cloudy skies

Four inconsistent snowfall generations from the bad input image, with the vessels displaced

Create a port backdrop on the right side of the image

Four inconsistent port backdrop generations from the bad input image, with the vessels displaced

Be Very Specific on the Changes¶

If you want to retain certain properties of the image’s environment, you need to specify them in the prompt.

❌ Environment properties left unspecified¶

add a giant volcano in the background

Generations from the unspecific prompt, where the water surface changes between images

✅ Properties to retain are spelled out¶

while keeping the water still, add a giant volcano in the background

Generations from the specific prompt, where the water stays still

Avoid Adding Subjects¶

Added subjects are not labelled. Also, if you try to add subjects behind existing subjects, they have a high chance of displacing the original subjects.

❌ Positions the new subject relative to existing subjects¶

Create a volcano in the distance, behind the ships

The volcano placed behind the ships displaces the original subjects

✅ Positions the new subject relative to the image¶

Create a volcano in the distant right side of the image

The volcano placed at the right side of the image leaves the ships intact

Avoid Conditionals¶

The model applies conditionals unreliably. This follows from the specificity rule: it sees pixels, not our asset mappings.

❌ A conditional prompt¶

If the ship is big, add a pink triangle on the top of the highest point of
the ship. If the ship is very small, add a green square on the left of the
ship

Four generations from the conditional prompt, each applying only part of the condition

In the above, we apply a conditional. Notice how each image applies only part of the condition: the prompt is only partially adhered to in every generation.