IT Cooking

Success is just one script away

ComfyUI Workflow: Coherent Facial Expressions V3.8

6 min read
ComfyUI Workflow: Coherent Facial Expressions generates a 3x3 grid of facial expressions, same character. Works for SD15, best with Anime models
coherentfacialexpressions3 featured meme

coherentfacialexpressions3 featured meme

Coherent Facial Expressions Workflow for comfyUI (aka Stable Siffusion) to generate a 3×3 grid of facial expressions, same character. Works for SD15, best with Anime models. Original author: auth0dev

The meat of this workflow lies in the area composition nodes. It relies heavily on the IPAdapter source image as you can see in the gallery. The new versions uses two ControlNet inputs : a 3×3 openpose faces, and the matching single openpose face. The graphic style and clothing is a little less stable, but the face fidelity and expression range are greatly improved. I even had to tone the prompts down otherwise the expressions were too strong.

coherentfacialexpressions3.8 openposegrid
coherentfacialexpressions3.8 openposegrid

 

Here are some highlights of what I modified and the json workflows to download:

  • v2.0: the original model from
  • v3.0:
    • Complete reoganization, cleanup, and standardized colors to make sense of this mess
    • IPAdapter bufix because of the latest update
  • v3.1:
    • tentative at auto-naming with KJNodes
    • Fast bypass added to enable upscale or not
    • Tile sizes auto-calculated
    • More then 60 links re-attached to allow a dynamic upscale size based off the input size
  • v3.2: auto-naming with KJNodes – mistake: do not install KJNodes, it causes conflicts
  • v3.3: Save Image Extended node developed thanks to this this workflow
  • v3.4:
    • auto-naming done the right way, with rgthree
    • model name text overlay
  • v3.5: added LoRA auto-bypass
  • v3.6: cleanup, ajustements, better file naming, infobox
  • v3.7: test with guy generation
  • v3.8:
    • 3×3 openpose grid is auto-generated
    • prompt weights reset to 1.1
  • v3.9:
    • generate openpose from input portrait
    • add prompt composer
    • variabilize the expression weights

 

Requisites for this comfyUI Workflow:

Custom Nodes:

With ComfyUI node manager, it’s no big deal: it will download all the necessary nodes for you. Here is the list nonetheless:

Models, LoRAs, Embeddings, ControlNets:

Models of your choice, but this workflow works much better with Anime, go figure.

Inputs / free samples

  • portrait: anything goes, square are better for IPAdapter
    portrait ginger girl 1
    portrait schwarzenegger 1
    portrait taylor 1

     

  • Openposes:
    openpose face portrait 1
    openpose face portrait 3x3

 

How To use this workflow:

  • Load the two openpose pictures in the corresponding image loaders: 3×3 and 1×1 :
    coherentfacialexpressions3.6 openposes
  • Load a face picture in the IPAdapter image loader: samples here, here and here
    coherentfacialexpressions3.6 ipadapter input
  • Select a checkpoint and vae loaders if you want to
  • Use the “Common positive prompt” node to add a common prompt prefix to all tiles
  • negative prompts can do more bad then good, beware

Stable Diffusion Sample Results

Anime Models

For Anime models, the expressions match the prompts quite nicely. Example below is done with Taylor’s portrait: note the similarities in the eyes, nose and lips. Even the hairs, that’s really crazy what IPAdapter is able to do

How much does the portrait influence the results? Here are the exact same outputs with the AI ginger girl. Sampler and clips do their best to match the prompt, but the IPAdapter weight = 1 is stronger

Realistic Models

For realistic models… not so much. Depends if the model has been trained and embeds a similar character as the portrait input. After all, you can only cram so much in 2GB.

What are these Artifacts?

They are caused by an exageration of the prompt weights for the individual expressions. I tried to crank them up to 1.3/1.5 to increase the results and that works at some extends, but also adds unwanted artifacts. Same models as above, same order, same exact prompts except all is back to the default weight 1.1, and you must admit it just falls flat. No artifacts tho…

Opportunity for Models Cleanup

This workflow has the added benefit to reveal crap models, and duplicate models. There are so much crap out there, developed for the sole purpose of one particular type of output the baker was looking for… Coherent Facial Expression helps you find models that miss character variety, or what the intent of the model really was, or what kind of training material was used…

When I started SD, I did not bother checking on civitai every time I got a model from huggingface. I just wanted to understand the workflow, how to alter it, discover and reproduce the results. Well, now that I try to get things done, it turns out some models were baked for certain purposes in mind, that were not told on huggingface. Whith limited set of characters ethnicity, suggestive poses, lack of facial expressions, lack of various outfit. Just for a very specific intent. Get it? Get it?

  • analogMadness_v50 clearly is a soft pr0n model. retired from citivai.
  • Deliberate_v2-inpainting another pr0n/furry/chinese model. retired from civitai.
  • f2222 is another pr0n korean model I saw a lot bqack in 2023. I didn’t bother checking on civitai at the time.

Reveal duplicate models

This worflow has another added benefit, to reveal duplicate models copies of each other, trained on same dataset: let’s skim!

Not by the same “bakers”, really? pixReal / fp16guy is the copy in this case. These are dupes, so I deleted pixReal.

Seriously… Now tell me those two are not the same base, and same training material… this is a joke.

Generalistic Models

How about the base models like SD1.5 v1-5-pruned-emaonly ? Well, forget about it: they are generalistic for a reason

cfe3 v1 5 pruned emaonly dpmpp 2m karras 24 6.5 0001
v1-5-pruned-emaonly

How About Men?

When you start getting things done, you hen realize some models where trained exclusively on chinese or korean characters, mostly females. Or the male characters they were trained on are just fashion models with a static bitch face. Completely useless… For certain expression they clearly only have trained on Korean pictures.

Can LoRAs improve anything at all? Nope, it all depends on what they are trained on. If they miss facial expressions, or trained only when the character has a certain age then it’s also useless. Just tested this Schwarzenegger LoRA. Actually it’s the only one. The other one available for SD15 is called Arnold Young and looks terrible. The one below clearly only embeds latest Arnorld, not the one from the 90s. Useless imo.

cfe3 epicphotogasm ultimatefidelity dpmpp 2m karras 24 6.5 swarz upscale 0013

Overall, I would say there is potential, but again what can you do when all you so-called “realistic” “best” “uber” “extra” models turn out to have all the same base?

How About mixing Openpose between single and 3×3 grid?

What if you mix different openposes between the single entry and the 3×3 grid? Well not much actually: some aspects of the 3×3 grid will differ but you cannot control which ones. In the examples below, only the bust and shoulders are affected:

coherentfacialexpressions3.8 openposegrid vary crop

 

Any ideas to make this worklow somewhat useful? Like detaching the tiles, or outputing individual tiles? I wonder why this was made the way it was. XY plot is more likely what should have been used in the first place. I smell lots of unneeded and extra work by the author. Stable Diffusion is so much fun!

 

Leave a Reply

Your email address will not be published. Required fields are marked *