Reverse Engineering Using Memory Inspection and Numerical Analysis

One of the more recent tools I’ve added to my repertoire for reverse engineering the game simulation is inspecting the memory of a running game to see how things vary without needing to pause the game and count individual tiles in the game many times.

Power Plants and Cheat Engine:

To calculate the MW rating for power plants to tiles powered I could have built a city and connected a power plant to it and then counted the number of powered tiles, but the game has a variable that stores the total power produced. I could just build a city and look at this value, swapping power plants out as needed.

My go to tool for finding, viewing (and sometimes editing) values in memory is Cheat Engine. I use it to search for a known value, pulled from the save file, and then I get the value at that memory location. SC2k always uses the same memory addresses each time it starts up, so I slowly built up a table of where memory locations end up, at least for the common stuff.

Why didn’t I just build the plant, pause, save, load it into my parser and look at the value? Memory inspection is way faster after the initial discovery, and allowed me to more rapidly test other things.

cheatengine values.PNG
Cheat Engine address table showing some of the interesting addressed. The total amount of power produced is highlighted.

Once I found the total power variable, it was pretty quick to calculate that the MW rating for power plants doesn’t actually mean anything and each power plant has an internal statistic for how many tiles it powers, including the tiles that make up the plant itself!
For more info, see the spec here.

Weather – Beyond Cheat Engine:

But what happens when I want to look at the correlation between various values over many game cycles? Does weather, specifically temperature, affect the crime rate? If so, how much? Weather definitely affects how much electricity solar and wind power plants produce as well as how much water is pumped, but what’s the formula there?

Basically, how do I get those values from the running game and make decisions on it?

I’m still working on answering that question, but at it’s most basic level, I want to read various memory values once per tick of the game. The tick count is also used to determine the date of the game, which starts at a certain epoch.

So I wrote a simple python program the dumps values at memory locations once per game tick and another that uses pandas, numpy and matplotlib to do some analysis.

For power plants, I didn’t need anything fancy. I could see by quick inspection of values that wind power is indeed strongly affected by altitude and to a lesser extent how windy it is. Solar power is also affected by how humid it is, but both also have a random component involved. The simulation specification documents show how the weather affects wind and solar power in greater detail.

But what does this “look” like? I logged several hundred thousand values and created a correlation graph using numpy, pandas and matplotlib and a city containing a single power plant and a single water pump.

power water correlation
Correlation between (solar) power and water production.

What’s this graph telling us? Well, solar power production is negatively correlated with how humid (-0.298) and windy (-0.245) it is. Not knowing exactly what this represents in the game, it seems that higher values are probably rainier. I didn’t bin the weather the same way, as the game treats it like discrete values rather than the “continuous” values for humidity and windiness. So solar power is definitely affected by weather. On the other hand, it looks like water production is positively impacted by the weather (0.486 correlation with humidity and 0.242 with windiness) which seems like in rainy weather pumps produces more water. This is about what the game does.

But what about crime? Below is a correlation graph generated the same way.

crime weather correlation
Correlation between crime and various game weather variables. The diagonal line are how much a thing correlates with itself, which is always 1.0 or completely correlated, as expected.

So what sort of correlation do we see between crime and how hot it is? Almost none, but there is a slight correlation, 0.0287, so it does affect the simulation a little bit. Unsurprisingly, the type of weather is well correlated with wind (0.525), humidity (0.877) and heat (0.287). This isn’t enough to know how the formula works, because there is randomness at work, but it’s a start to figuring it out.

I’m not sure if this sort of analysis is actually going to be useful working to reverse engineer the game, but it’s definitely a useful place to get started, and may yet be a useful tool to determine exactly what is happening internal to the simulation.

 

Advertisements

Thomnar – Multiple Tile City

One of my overarching goals for a re-implementation of SimCity 2000 is making city sizes much larger than the 128×128 tile minimum. This is based on one city I made where I built a 25 tile city in a 5×5 grid by manually reconciling the edges. Here’s a little more about that city. The city’s name, Thomnar, doesn’t mean anything but just sounded good.

The final result represents somewhere around 100 hours of work spread over 8 years while I was in high-school and University. Most of this was spent reconciling the edges of the maps, building out the actual cities and then a little bit in Photoshop stitching the screenshots together.

When building it, I tried to keep in mind how many North American cities looked. A dense core giving way to poorly planned sprawl, lots of horrible land use around roads, and true wilderness on the outskirts. Lots of power plants and other services to support the city, but given I have no way of moving power/water between tiles, each tile did need its own power and water utilities even if adjacent plants would be enough to power it.

thomnar-5x5.jpg
Thomnar, in its current full glory. Current population, 833,010 sims.

The first tile is actually based on the Charleston scenario city. When I finished building the city, looked like half the city was missing, so I added another tile on top to balance things out and make it look better.

thomnar-teaser
The second, continuation tile is at the top. Population: 250,760 sims.

Next I added some more tiles on the edge, which also looks like the view from the neighbours window.

thomnar neighbours
The neighbour’s window in game.

 

thomnar-teaser2
What I imagine the city’s neighbours would look like. Population: 395,900 sims.

Next I filled in the corner tiles to get a 3×3 grid, and then decided that the city looked weird without surrounding towns and wilderness, which resulted in this 4×5 grid.

thomnar-teaser4
4×5 grid of Thomnar. Population: 748,920.

From the 4×5 grid, I added another set of tiles to get the full square 5×5 grid. I created another couple of tiles that aren’t included yet, but by then, reconciling edges was starting to feel like a chore, so I stopped work on the city.

Gallery of some of the neat details I added to the city (if anything looks quirky, it’s because the images are rendered outside of the game using in-game assets):

Things SimCity 2000 Never Intended You to Do

This post is meant as a showcase of some of the proof-of-concept things I’ve done with SimCity 2000 that it had never intended you to do.

The way the game stores an in progress disaster is by setting the city to disaster mode and setting the disaster type in the MISC segment of the .sc2 file and then adding actual disaster tiles to the XTXT segment/layer. But what happens if we don’t go into disaster mode and place disaster tiles? Well, we get a city floating in clouds, and hell, a city that’s eternally on fire but never consumed.

And once you can change anything at will, there are other interesting things that can be done. Make mixed bridges? Easy. Edit the terrain after the city has been built? Yep, that’s a single value changed. Make tunnels cross? Well, sort of. The game doesn’t seem to know what to do with them. Avenues? Yep, just remove the intersections.

There’s a lot more possible, but I’ve been focused on getting more of the game reverse engineered over building better editing tools. I have considered building a standalone editor that would also function as a game frontend, but haven’t had time to do much with it.

I’ve also used the editing ability to make cities that have features some cities in real-life have. Rivers in many places flood, and often have levees along them, so I made a city that used the completely raised terrain tile as levees along a river instead of making a 3-tile wide strip of raised terrain.

levees.PNG
Levees around a city to protect a city from flooding. Normally the game wouldn’t allow you to do this, but it adds some realism, especially with the non-ramp access to the bridge.

SimCity 2000

Featured

SimCity 2000 is the second of Maxis’ well received city builders and is my favourite in the series, in addition to being one of my favourite games of all time. I’m not sure exactly why, but it’s the best blend of building and management and the art.

I found that SimCity 4 was all about micromanagement at a scale that wasn’t fun. The game was too hard, at least without mods to fix some of the more serious issues. The art style is very well done 8-bit graphics which I knew weren’t supposed to be hyper-realistic and never looked janky, unlike some of the later games. Maybe there’s also some nostalgia too.

Despite being released around a quarter-century ago as of this writing, I still play the game occasionally. I play the Windows 95 version, which is the definitive version. Unfortunately, it doesn’t run on modern versions of Windows without a patch to the executable, so the inferior DOS version is the only one commercially available presently.

Even when I was a child, playing the game, I’d often get to the (admittedly large for the time) limits of the 128×128 tile map and wish my city could somehow continue past it, but actually doing so was beyond my abilities then.

I’d built one city out of 25 individual tiles over several years during high-school and early University, painstakingly reconciling the edges as I went. It was slow going, and I eventually got tired of it as it was getting to be more and more work as it grew.

Thomnar 5x5
A city I called Thomnar, which doesn’t mean anything, that I built using 5×5 city tiles.

And then in 2014, I was finishing up my last semester of University and working, and I felt the draw to the game again. I still really liked that large, multi-tile city and wanted to work on it more, but reconciling the edges was annoying. Except this time, I had the tools and technical ability to start reverse engineering the game and working on a re-implementation.

My initial goal was simple: figure out the city binary file format and write something to automatically reconcile the edges.

As I figured out more and more of the format, my focus shifted from writing a tool to reconcile edges to a complete re-implementation of the game. This also involved figuring out the complete sprite file spec, as well as working on the text data files.

floating cloud city
Having the (nearly) complete city file spec also means that I can make cities that are impossible to make in the game, such as this city that is floating in the clouds.

I have gotten most of the .sc2 file spec figured out, all of the game sprite file spec as well as the .mif sprite spec and some of the text file specs done, which are kept up-to-date on my GitHub.

I also have written a Python library and related tools to open, edit and save out the modified .sc2 file, but it isn’t ready for public release yet. When it is, I’ll update this post.

SimCity 2000 Posts:

Things SimCity 2000 Never Intended You to Do
Thomnar – Multiple Tile City
Reverse Engineering Using Memory Inspection and Numerical Analysis

Useful SimCity 2000 Links:

Run the Windows 95 version on modern Windows
OpenSC2K – Open Source re-implementation
Pat Coston’s ClubOpolis
My documentation on the game

 

Not NaNo Retrospective

Things went well at the start of the month, and I got a lot of work done on the game. In the second half of the month, things started taking longer and started requiring more knowledge of how cocos2d works. I never felt like I understood how exactly cocos2d’s event model worked, because I never did figure out how to have multiple things moving in the game at the same time. There seems like there should have been a way to set up a “do-x-per-tick” way to do events, because it’s common in games, but I never figured out how to do it.

And while I was struggling to get this figured out, I also got sick, which dramatically curtailed my ability to work on the game after the 22nd or so.

I enjoyed working on the game, and I think that doing it sort of like NaNoWriMo also worked well.  One of the big differences I realized is that it’s not really possible to get stuck in NaNo and not know how to proceed. Coming up with a story doesn’t require the tools to be properly understood (assuming you’re not doing it in a text editor with obscure command such as vim or emacs as well as writing), so there’s not really a way to get stuck. Even if the story isn’t going anywhere, you can always move on to the next story.

I think cocos2d is significantly better suited to doing 2D work than Panda3D, which is reasonable, given their focus. I’m glad I didn’t end up using Panda3D. However, cocos2d doesn’t seem to have many examples, the docs are very barebones, and it hasn’t been updated (as of when I wrote this) in 14 months, which is starting to look like an abandoned project.

Would I do it again? Probably. We’ll see next year.

Not NaNo Day 22

I fixed a small bug that kept enemy units from getting targets to attack. It needs some more work, because now the paths generated will go into a core’s safe area, which is something I’d like to avoid. I’ll probably have the enemy choose a different target if this happens, or wander aimlessly/despawn if there aren’t any valid targets.

But I ran into another problem, one which I already sort of knew existed. Two things don’t seem to be able to move at once. They technically can, but there are some weird issues around it. I don’t know if it’s because it’s happening in a subprocess (I’m not sure how cocos2d works this way) or what, but I’ve been unable to figure out exactly how to fix it. Ideally, I’d like some global tick rate, where things are updated every tick. Slower things might be updated some number of ticks, but the concept is the same.

I’m not sure how to do this with cocos2d, or even if it’s possible to do with cocos2d. I spent quite a bit of time trying to get something working, and not coming up with a good solution, so today’s update is a bit light.

cocos2d enemy attack.gif
Enemy appearing to attack a unit. It’s a fluke, not the intended behaviour. In the game it also doesn’t cycle, it goes the one spot and stops.

Not NaNo Day 21

I decided to tackle some of the performance issues today. First, I fixed the A* implementation to match the fixed pathfinding implementation. This didn’t resolve my performance issues, and I wasn’t expecting it to, but I wanted to make sure that it wasn’t my issue.

What I thought was my issue was the code that checked network connectivity. I timed that segment of code, and to my surprise, it was running in under 1ms, so that certainly isn’t why the game is crawling along when lots of energy network segments are being checked as powered or not.

That leaves the actual drawing function. With one network segment, it takes ~1ms again do draw, but by the time even ~15 are on the screen, it’s taking ~10ms, and when 85 are on the screen, ~75ms. Still not the smoking gun I expected, but maybe the issue was elsewhere in drawing above the layer where I am technically setting the sprites up to draw.

And then I noticed that I’d missed clearing the children of the network layer. I’m not sure why I need to do this, but I’d already figured out that it seriously impacted performance elsewhere and had to do it. What I’d missed doing for the network and safe layers, leads to huge amounts of sprites that are being drawn, just not visibly (I think, cocos2d/pyglet can be a little opaque at times). Each time a batch is added, it stays there unless it is explicitly gotten rid of, even things that are updated many times. I still don’t know the proper way to do this, but making the change made the game run about 5-10x faster.

cocos2d fps boost.PNG
8 FPS isn’t terrific, but it’s sure better than 0 FPS.