Modding Age of Empires II with a Sprite-Diffuser

Mai 21, às 23:14


7 min de leitura


1 leituras

Last month I set out to create a way for my friends to make custom civilization sprites for our Age of Empires II lobbies.Below are some thoughts and process on how to create a versatile prom
Modding Age of Empires II with a Sprite-Diffuser

Last month I set out to create a way for my friends to make custom civilization sprites for our Age of Empires II lobbies.

Below are some thoughts and process on how to create a versatile prompt-based image generator. For beginners I would recommend Alpaca, and for those comfortable with coding – Stable Diffusion Web UI and Python.

Special thanks to the AoEII modding communities OpenAge, SLX Studio, and Age of Kings Heaven.

Visual Explorations

Screenshot Transfers

This idea came about while playing with image transfer (img2img) on Age of Empires II screenshots. All the techniques described below use the Stable Diffusion generative “AI” (which I also refer to as the model or network).

I started in Photoshop with Alpaca and worked on full game screenshots. Mostly what I was looking for was how the model behaved, what shapes and textures it chose to preserve and what it discarded in its stylistic wanderings. Overall it managed to stay in perspective and theme quite well.

Age of Empires II in anime, Martian, painterly, and other styles.

What was immediately interesting was how elements of the image started working together. Tiled terrain gave way to roads and paths – there is communication between the buildings and environment. For now we will only be working with building sprites – but it demonstrates that machine learning could be a good tool for blending procedural elements of a game.

The Caspar David Friedrich Benchmark

Following this I prompted the network to output images into a strongly isometric perspective, looking to get consistent results across lighting, color, shape, and texture.

Another screenshot but experimenting with different seeds and weights.

Some really awesome results, as the strength of the image transfer increases, parts of the minimap and UI would become other buildings or terrain.

Anime Loyalists vs. Moon Colonists vs. Zombie Romans

Now it was time to make some civilizations.

When generating the model requires 512×512 images, luckily all the Age of Empires II sprites are about half that so we don’t have to worry about resizing. These were the first results.

These buildings were cherrypicked and cleaned in Photoshop and weren’t batched. In order to get diverse results the strength of the image transfer was set very high, but as a result the lighting, proportion, and level of detail varied greatly. The castle asset (left) would continue to be a pain point as it is 2x larger most buildings, but the network would insist it was an oversized house.

Refining Outputs with Control Net and Loopbacks

To get consistent results while having a high transfer strength I needed to use Control Net, a system that guides generation using input like lineart, depth, or segmentation.

At this point everything worked, sprites stayed in perspective and listened to the prompt, but they failed to be imaginative or unexpected. This can’t be solved by making the prompt strength extremely high, the lighting or color will start to vary too much, control net cannot help there.

Instead, there is a technique called loopback – this runs the generated image back over itself with the same prompt. Without it, the network isn’t able to imagine novel changes to shapes, textures, and color.

Novel shapes emerging in the interior of the shape thanks to looping.

The final technique I settled on was Control Net with Zoe Depth Estimation, two loopbacks, both with about 85% strength (only 15% of the original image is preserved).


This is arguably the most important part of the process, but I have left it for last. That is because copying what worked for me will probably hold you back, I didn’t spend that much time on prompting! But this is a good place to start.

I used simple formula for each image with the following prompts. The only field that changed for each building was the Subject (i.e. Archery Range, Market, House).

  1. Shape
  2. Descriptor
  3. Subject
  4. Style
  5. Emphasis
  6. Modifiers
  1. Isometric exterior of a
  2. ancient Roman
  3. Barracks
  4. in the style of Giovanni Paolo Panini
  5. 3D roman architecture, greco-roman stone and pillars with intricate stonework and roofs
  6. desaturated, 8k, bright sunny natural lighting, trending on artstation

Lastly, Stable Diffusion does not create transparencies. I thought object detection would work but surprisingly they struggled to find a solid mask. Instead I forced a solid background color in the prompt and created a few transparent flood fills with ImageMagick

Final Result

Other Experiments and Thoughts

Custom Buildings & Fine Tuning

It also possible to reverse the process. Instead of creating different styles of existing sprites, new sprites in the original style could be made using a fine-tuned Age of Empires II model. Either 3D blockouts or photo images of real buildings could work as inputs.

  • Image gallery image

Greyboxing to Image

With one of my existing Unity projects, I took a greyboxed level and applied segmentation to the buildings, ground, and trees according to the ADE20k dataset.

  • Image gallery image
  • Image gallery image
Original image, object segmentation map, and outputs.

With Control Net and object segmentation we can run it through the same Caspar David Friedrich prompt. This could be used in concept art or level design workflows to quickly block-out environment art.

Image Rich Mindset, Seeding, Spawning

With so much semantic and syntactic information of an image now available, each image in the project can be a starting point for synthesizing new outputs.

Recreate Age of Empires II inside the model with only a depth map and no screenshot for reference.

By defining 10-20 hero assets using a traditional art workflows, be it characters, buildings, or environments, a development team could then kit bash new assets together. Studios with a large internal catalogue of concept art and assets might be interested in bringing this to fans to extend their lore and worlds.


These models are surprisingly versatile and are a lot of fun to work with. Future games, should they choose to, could create a set of base assets that then seed user-generated lore or internal development. The game world is itself an image model, video games are both a place to be and a desirable frame for future images.

The project is online at @neilsonks and

Continue lendo


Como corrigir problemas do sistema iOS (iPhone/iPad)
Não se desespere ao se deparar com problemas do sistema iOS em seu iPhone ou iPad. Com um software simples e poderoso, você consegue corrigir a maioria deles sem precisar de assistência técnica.

Mai 30, às 22:19


12 melhores sites com cursos de tecnologia gratuitos
Índice Como escolher bons cursos gratuitos?CourseraedXUdemyLearnCafeFGVRecodeUdacityGoogle Developers TrainingMicrosoft LearnFreeCodeCampMoodle IFRSOmie.AcademyCompartilhe e amplie o poder da educação...

Mai 30, às 21:08


Share Your Expertise & Stories with #devpride this Pride Month! 🌈
Happy Pride, all! ❤️ 🧡 💛 💚 💙 💜 🤎 🖤 🤍 💗 We’re so inspired by the contributions and engagement within the queer community on DEV. We regularly witness creativity and expertise shining through on posts...

Mai 30, às 20:43


20 sites de Inteligência Artificial (AI) que mudarão sua maneira de trabalhar
Índice LucidPicPoisedMurf AIAutodesignerTalk to BooksFROMAGeKaiber StockImgBardeen AIPowerMode AIControlNetAI Image UpscalerFormula GeneratorDeepAIPDF.aiRunwaytldv.ioSoundrawSnack PromptFree TTS Com a...

Mai 30, às 20:10

AI | Techcrunch

Meet the Aerospace Corporation, AMD, Torc and more at Disrupt
TechCrunch Disrupt, a veritable who’s who and what’s what in the startup world, takes place on September 19–21 in San Francisco. It’s the perfect destination to find inspiration, gain knowledge, forge new...

Mai 30, às 20:00

AI | Techcrunch

SEC settles with former Coinbase employee over insider trading charges
The U.S. Securities and Exchange Commission (SEC) has settled charges with a former Coinbase product manager and his brother for engaging in insider trading, the agency announced Tuesday. Ishan Wahi, the...

Mai 30, às 19:49

AI | Techcrunch

Google DeepMind introduces Barkour, a benchmark for quadrupedal robots
The bipedal humanoids may, in fact, be coming — but the quadrupeds are already here. They’re in labs, doing inspections in power plants and refineries, playing soccer and are even — much to the concern of...

Mai 30, às 19:20


Teste de Software: Garantindo Qualidade em Todos os Estágios · arthurborgesfs
No mundo acelerado do desenvolvimento de software, garantir a qualidade e a eficiência é fundamental para o sucesso de qualquer projeto. Uma das etapas cruciais nesse processo é o teste d...

Mai 30, às 18:55

AI | Techcrunch

The academic to founder pipeline with Dr. Stacy Blain from Concarlo Therapeutics
Welcome back to Found, where we get the stories behind the startups. This week Becca and Dom are joined by Stacy Blain, the co-founder and chief science officer at Concarlo Therapeutics. Blain talked about...

Mai 30, às 18:49

AI | Techcrunch

Apple WWDC 2023 is next week -- here’s what we expect
Yes, the headset -- some iOS, MacOS and watchOS announcements, too Look. We’re all interested in one. I know it, you know it, Apple certainly knows it. Much as AI was Google’s primary focus at...

Mai 30, às 18:45