Skip to content

Why doesn't AI follow my sketch?

Usually because the drawing leaves something open, and the AI fills each gap with what buildings usually look like. Dark, closed lines in one perspective, with nothing extra on the page, leave it far less to guess.

By the Sketchfest team9 min read

A row of ink sketches of a mountain hut pinned on a studio wall, each with a printed render below it, and an architect looking at them

You draw a steep roof and the render comes back with a shallow one. You draw one door and get two. It happens a lot: in the 2026 Chaos and Architizer survey of nearly 800 architects and designers, 48% named inconsistent or poor output quality as their biggest challenge with AI. Often the drift starts on the paper, which is also where it is quickest to fix.

Below: why AI drifts, a symptom table and a checklist of six common sketch faults. For the whole route from a blank page to a finished image, start with how to turn a sketch into a realistic render.

Why AI drifts from your drawing

Think of the predictive text on your phone: type "see you" and it suggests "tomorrow". An AI image model does something similar with pictures. Wherever your drawing stops saying something, it fills the gap with the most usual answer. Leave the roof pitch unclear and you get a usual roof.

So a render drifts for one of three reasons:

  • The drawing leaves it open. A faint line, a gap at a corner or a blank wall is a question, and the AI answers it with the usual building.
  • The drawing says two things. Lines aimed at two vanishing points, or a dimension line that looks like a railing. The AI picks one reading, and it may not be yours.
  • The words lead. In general image tools, your text steers the image and the drawing guides it.

What the research shows

ControlNet, a 2023 method by Lvmin Zhang, Anyi Rao and Maneesh Agrawala, makes an image model follow an extra picture, such as edges, depth or a scribble. Two later studies asked which drawings such models follow best.

In 2024, Sucheng Yao and Kanjanee Budthimedhee of King Mongkut's University of Technology Thonburi in Bangkok ran three architectural sketches, from loose freehand to fully ruled, through five ControlNet models. They found that it "performs better with orderly hand-drawn sketches" and that "minimizing conceptual lines in hand-drawn sketches reduces the likelihood of rendering failures". In their words, "Freer linework leads to more varied architectural results."

Subhadeep Koley and six colleagues studied freehand sketches in general, in a paper at the CVPR 2024 computer vision conference. Models of this kind, they write, "directly translate sketch contours into the output photo domain", so the wobbles of an amateur sketch come back as a deformed object.

Both studies are limited: the Bangkok one used three drawings found online and described its results rather than measuring them, and the second was not about buildings. But both point the same way: clearer lines are followed more closely.

Why general tools lean on your words

ChatGPT, Midjourney and Adobe Firefly were built to make images from words. All three take a sketch too, but their makers' pages (checked 27 October 2026) show the words leading:

  • ChatGPT. In the mobile app you can "draw a sketch and use it with written instructions to create an image".
  • Midjourney. Image prompts work "as inspiration to guide new creations, not to copy them exactly". With one image you also need a text prompt, and an image weight setting decides how much the image counts.
  • Adobe Firefly. It asks for "a detailed description of the image you wish to create". A composition reference can match your drawing's "outline and depth", with a Strength slider for how closely.

Open models such as Stable Diffusion show the same balance. In the Diffusers library, an image-to-image strength of 1 "essentially ignores" your image, and a higher guidance scale ties the result more closely to the text. So drift can come from a sentence or a setting, not only from the drawing.

Problems, causes and fixes

What you seeLikely causeFix
It changed my roofFaint or short ridge and eavesRidge from gable to gable, eaves to the corners
It added a floor or windowsA tall blank wallLine up each floor's windows; give a blank wall its material
It moved or lost the doorA bare rectangle, or a small door on a far wallA frame, a step and a path; a view that shows the door
The building is the wrong sizeNo scale clueOne standing figure, head on the horizon; size the door to it
It ignored my materialsLines show shape, not materialA small patch of each material, and the line where they meet
It invented surroundingsBlank paper leaves the setting openDraw the ground, the slope, the path and the key trees
It still looks like a drawingShading or grey paper; a high control weight in open toolsClean lines on white; ask for a photo; lower the control weight
Every try gives a different buildingEach try starts from new random noiseList what must not change; redraw the part that fails most
It followed my words, not my drawingIn a general tool, the prompt leadsDescribe what you drew; raise the image weight or Strength

On size, the Bangkok researchers noted that their generated images "lack a sense of scale". For an eye-level view, David Drazil's Sketch Like an Architect blog gives the classic rule: "place all the standing people's heads on the horizon line".

On the drawing look, the Bangkok study found that once one model's weight reached 1.0, "the line elements in the images are intensified", and Koley's team note that a higher sketch weight "almost always produces deformed and non-photorealistic outputs". The Bangkok study suggests keeping it "between 0.5 and 1.0".

On variety, the Hugging Face Diffusers guide says: "Diffusion is a random process that generates a different output every time." In open models a fixed seed keeps tries alike, but it does not make them follow your drawing more closely.

Six common sketch faults and how to fix them

Check your drawing against these six before you render.

  1. Faint pencil. Light, grey lines are easy to lose, so parts of the outline can go missing. Fix: go over the outline, the roof and the openings in black ink, and rub out the pencil.
  2. Gaps and wobbles. An open corner or a broken ridge leaves the shape unfinished, and a model that follows your lines closely follows a wobble too. Fix: close the outlines that matter, draw long edges in one steady stroke, and keep one line per edge.
  3. Two vanishing points for one façade. In perspective, "Parallel lines of parallel planes converge at the same vanishing point", as the University of Delaware's drawing guide puts it. If the roof aims somewhere else than its walls, the AI tries to follow both. Fix: extend each façade's lines lightly with a straight edge, check they meet at one point on the horizon, and redraw any that miss.
  4. Notes on the drawing. The AI reads lines as the edges of things. A dimension line can become a railing, a leader arrow a post, and handwriting can come back as lettering on a wall. Fix: crop notes away, or keep them on tracing paper over the drawing and capture the drawing without it.
  5. Heavy hatching. The AI cannot tell shading from material, and it tends to read strokes as joints. Dense cross-hatching can come back as a dark material or an opening. Fix: hatch only for material, a small patch on each surface, with strokes that follow its real joints. Leave shading out; the render makes its own light.
  6. A busy background. Trees, hills and clouds drawn as firmly as the building can become part of it. A mountain ridge behind the roof can turn into a second roof. Fix: draw the setting lighter and simpler, and leave a strip of white paper between the building and everything behind it.

If the drawing passes all six and the render still bends, check the photo: flat, evenly lit and sharp. In Midjourney, a new view can even be the aim: its Edit model, for versions 8.1 and 8.2, is there to "Shift the perspective and show different views". For a one-page version of the drawing rules, see how to prepare a hand sketch for a clean render.

Where Sketchfest fits

In ChatGPT, Midjourney and Adobe Firefly your words lead, so you describe the building and check how much of your drawing came back. We compare them in which AI is best for architectural renders.

Sketchfest is built for architects and works the other way round. You upload a hand sketch, a CAD screenshot or a 3D view and get a photoreal render of your design in seconds. It follows your drawing and there is no prompt to write, so your lines lead and the AI fills in the rest. There is nothing to install, it works in your browser, and your projects stay private.

RenderHand sketch
RenderHand sketch
A hand sketch of a house on a hillside with wide front steps and its render, made with AI in Sketchfest. Drag the line to compare.

Quick answers

Can AI follow a rough sketch?

Yes, for the parts that are clear. Dark, closed lines for the outline, the roof and the openings give the AI the design. In the Bangkok study, orderly sketches were followed more closely than free ones.

Will a CAD screenshot be followed more closely than a sketch?

It often is, because a CAD view is orderly: straight lines, one perspective, closed outlines. The Bangkok study found that sketches made with drafting tools, with fewer extra lines, gave more accurate images. Hide dimensions, hatches and grids first. For model views, see how to render a SketchUp model.

Why does ChatGPT or Midjourney change my sketch?

Both make a new image from your sketch and your words, so whatever the words leave open, they decide. Describe the building as drawn: the storeys, the roof form, where the door is. In Midjourney, raise the image weight. Then compare the result with the sketch, part by part.

Sources

  1. Chaos and Architizer, The state of AI in architecture (nearly 800 architects and designers, 2026; run by a software maker; page last updated 23 March 2026)
  2. Zhang, Rao and Agrawala, Adding Conditional Control to Text-to-Image Diffusion Models (Lvmin Zhang, Anyi Rao and Maneesh Agrawala; ControlNet, 2023; version 3, 26 November 2023)
  3. Yao and Budthimedhee, From Sketches to Renderings: A Comparison of Rapid Visualization Methods for Sketches Based on ControlNet (Sucheng Yao and Kanjanee Budthimedhee, King Mongkut's University of Technology Thonburi, conference paper, 2024; three sketches found on Pinterest, results described, not measured)
  4. Koley, Bhunia, Sekhri, Sain, Chowdhury, Xiang and Song, It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (Subhadeep Koley, Ayan Kumar Bhunia, Deeptanshu Sekhri, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang and Yi-Zhe Song; University of Surrey; CVPR 2024, pages 7204 to 7214; quotes checked in arXiv version 2, 20 March 2024; freehand sketches in general, not architecture)
  5. Hugging Face, Diffusers: Image-to-image pipeline (strength and guidance scale; version 0.41.0; checked 27 October 2026)
  6. Hugging Face, Diffusers: Reproducibility (random noise and seeds; checked 27 October 2026)
  7. OpenAI Help, Images in ChatGPT (sketch with written instructions; checked 27 October 2026)
  8. Midjourney Docs, Image Prompts (inspiration, not copies; text prompt with a single image; image weight; checked 27 October 2026)
  9. Midjourney Docs, Edit Model (versions 8.1 and 8.2; shifting the perspective; checked 27 October 2026)
  10. Adobe Help, Generate images from text prompts in Firefly (prompt first, composition section; updated 9 July 2026)
  11. Adobe Help, Match image composition to a reference image in Firefly (outline and depth, Strength slider; updated 16 June 2026)
  12. University of Delaware, Art Foundations: Linear perspective (drawing guide, undated; checked 27 October 2026)
  13. David Drazil, Sketch Like an Architect: 3 common mistakes when drawing people (blog, undated; heads on the horizon line in an eye-level view; checked 27 October 2026)

Try it on your own drawing

Drop a hand sketch, or a CAD or 3D screenshot. Your first test render is free.

See pricing

Keep reading

Drop a sketch. Get a render.

The AI render tool made for architects.

Explore Sketchfest