As someone else pointed out, there are systems where you can isolate and replace only a portion of the image while leaving the rest unchanged, but you’re correct in this case, as you point out in your last sentence.
This sort of system takes the input image which it blurs and adds noise, and then regenerates from that rather than an initial pure-noise seed. You can mask or add more noise to the areas that you want and more dramatic change. Whole image-to-image diffusion; mostly uses the same mechanisms as a regular text-to-image diffusion.
As someone else pointed out, there are systems where you can isolate and replace only a portion of the image while leaving the rest unchanged, but you’re correct in this case, as you point out in your last sentence.
This sort of system takes the input image which it blurs and adds noise, and then regenerates from that rather than an initial pure-noise seed. You can mask or add more noise to the areas that you want and more dramatic change. Whole image-to-image diffusion; mostly uses the same mechanisms as a regular text-to-image diffusion.