Finally, after 10 months, I am able to save and load my game world! Prior to this, every day, every time I ran the game, the game world had to be generated on the fly. Even when I was doing live demos at conferences and twitch. I suppose the nice side benefit of this bug undocumented feature is that my world generation AI is rock solid and fast; it’s probably been tested 10,000 times. :-)
Back when I wrote the prototype in WinForms .NET C# the game world did save. Everything was serialized through XmlSerializer and then later BinaryFormatter. These worked fine and confirmed my proof of concept worked. But when I took that approach into Unity it crapped out. There are differences between the Microsoft .NET runtime and the Mono runtime of Unity and serialization is one of those areas where not everything works the same.
Rewriting data serialization is not sexy work. You can’t post pictures of it on the website. It doesn’t show well in a trailer. With so much other work to be done, I put it off.
“Redoing data serialization is the programming equivalent of cleaning your toilet:
It is obvious when you have to do it.
The longer you put it off, the worse it gets.
You will never “feel” like doing it. You just have to do it.”
It has taken me about a month to rewrite and test my data serialization. I’ve learned some things the hard way I sure wish someone would have shared before I started. This article is a post mortem of Unity data serialization. As far as the game goes, I’m so sick and tired of rewriting stuff I’m going to add some new features this week! Any new features will be better than what I just went through! :-)
I tried very hard to make an interesting infographic, but this is the best I could come up with. The bar on the right is the number of class files by type I had to change. Due to inheritance the buildings went surprisingly easy.
Simple vs Complex Data to be Serialized
Games vary vastly in the state data they need to serialize and retrieve. Super Mario Bros. U is the most complex Mario to date, yet all the data needed to store player progress can be done in a simple list of variables like: lives remaining, current coins, current level, completion flag for each level, perhaps a state flag for each bonus house. Pretty basic stuff. Many FPS will fall into this category as well: current level, rounds of ammo, flags for the weapons you do or do not have, and a 3d coordinate of where the player is standing. There could be lots of data to save, but it is simple data not complex data. It can be expressed through simple data types like numbers, strings, and Boolean flags on a one to one basis.
Now take a game like Skyrim, where (seemingly) every object you’ve ever encountered is saved: location, rotation, and more. Then there is all the dialog trees and attitudes of every character you’ve ever met. The interrelation of objects and characters is what makes it complex. There isn’t even necessarily a fixed list of things to save/load. This is similar to what Archmage Rises is like, but by going one step further: world generation.
Skyrim is great, and they do have to store world data of each character, tree, and road. But that data is created by designers and read into the game world at run time. Your Skyrim game world (where a city is, or a road is placed) is the same as my Skyrim game world. This means THAT kind of data does not need to be saved as part of the state serialization.
But Archmage Rises generates a unique world for each player. That means the serialization has to save and load not just the shop keeper’s items and prices, but also where the shop is, where the shop’s town is located, and by which roads you can even get to this town. Since each town is generated as part of the GM AI big bang, I don’t even know what buildings may be in what towns. Then there are all the NPCs, with their own unique agendas and relationships with other NPCs. A milk maid may only know her family, where as an inn keeper may know everyone from here to three towns over.
This kind of data cannot be stored as a simple list of numbers and strings. I need a solution that can store custom data objects of any design, both known now and unknown ones I will create in the future.
Finally, I also store some data (tradable commodities for instance) in Dictionaries (keyed lookups). Dictionaries are notoriously hard for serializers to handle. Many won’t handle them.
The Multiplying Wives Problem of Serializing References
Say you have an NPC like a guard (G) who works in a particular town. He has a lovely wife (W) but a bossy Captain (C). The generic Character class can look like this:
Public class Character
{
public string Name;
public Jobtype job;
public List<Relationship> relationships;
}
And a Relationship can simply look like this:
Public class Relationship
{
Public Character targetPerson;
Public float feeling;
}The guard object will have two entries in his relationship list:
- Targeting his wife with, hopefully, a high feeling value like 100
- Targeting his boss with a mediocre feeling value of 40
But relationships are two way streets. The wife also has a relationship with her husband the guard, so she will need to store a reference to his character object with her feelings for him (which are not necessarily the same as his feelings for her, which makes for interesting emergent stories).
Now, with this is a really simplified example of only three characters, if we use standard .NET serialization what will happen?
It will save all public fields and referenced objects to a file. But when you go to load them something very bad will occur. The serialization will create complete atomic objects not references to shared objects.
I will try to explain why:
The guard object G will have two references in his relationship table: one to Wife W and one to Captain C. But when the world recreates the wife character it won’t marry up the guard G and the wife W objects (nice pun eh?), it will instead assume the wife is brand new and create a new wife object (W2) with all the correct values on it. There will now be two instances of the wife in the game world: W from the guard and W2 the actual wife. And there will be two instances of the guard as well! The actual guard object G who has a relationship with the wife W, but then the copy of the guard, G2, which has a relationship coming from wife W2.
The game will play perfectly well at first. There won’t be any errors or crashing, because the game will look identical to the state before the save. But as you play subtle issues will start cropping up. Two guards with the same name will be in a town. Unlikely given the world generator code, but not impossible. But subtler things like if Guard G is killed, wife W will be in mourning but wife W2 won’t, because her copy of the guard never died. As the player encounters the wife they will think she is schizo, sometimes mourning and angry about her loss, other times oblivious to it. The issue is the multiple copies of what should be the same object reference.
Looking for a Solution
I searched around for solutions: asked other devs, searched forums, asset store, etc.
My ideal solution:
- Solves object reference issue
- Works with all my existing data types, collections (including dictionary), and unity types (if I need it too)
- Works inside and outside of unity. I want to use the same solution for my content editor (BARD) as I use for the game. Being able to test both inside and outside (C# WinForms or Command Line app) of Unity allows me to work faster when my current Build->Play time is around a minute.
- Simple to use: just feed it an object to serialize, and it crawls through saving all the child objects.
- Minimizes code rework. I use entity classes with public fields to represent everything.
Some of the things I looked at are:
Unity Serialization Best Practices Megapost
I was going to write some detailed remarks on each of those solutions, but since I didn’t actually use any of them my comments would probably be wrong or misleading. So instead, I’ll just say why I didn’t choose them.
Either their documentation specifically stated a scenario they couldn’t serialize that I wanted, or it was unclear from the documentation/examples that it met my needs, like working outside of Unity.
When Stuck, Hire a Fan!
I knew what I wanted for a solution, but it was going to take some serious time to evaluate my options and get something working. Fortunately at around the same time a fan, Don Bloomfield, volunteered to help out with the game. After evaluating his skills I decided to let him chew on this very precise problem and see what he came up with.
Solution: SharpSerializer
Don took a few sample entity classes and got a solution going with the open source (free) SharpSerializer. He gave me back his project solution and I was able to follow it and apply it to the game.
It met all of my requirements except for minimizing code rework. I guess it could be worse, but because I used public FIELDS instead of public PROPERTIES for everything I had to go through changing it all over. I also had to add public parameterless constructors to many of my objects and redo all initialization.
Finally, when searching about the integration between Unity and SharpSerializer I stumbled across a bug report that shows it’s the solution Obsidian (Pillars of Eternity) uses. Hey, if its good enough for the big guys, it’s probably good enough for me!
So now that I got it working, these are the lessons I learned.
Lesson 1: Have One Object That Holds World State
I got this one right from day one, yeah! I created a singleton called Game that stores everything related to the game world. Lists of towns, roads, characters, players current gold, date, everything!
This approach made it extremely easy to farm out the serialization work because Don only had to work with one class (of course there were many custom child object classes within it but those were easily commented out).
It also makes it easy to code. As a singleton, every object can see it and set/check its values. It’s also easy to code the serialization: can the Game object save? Yes. Can it load? Yes. Then we’re done! One world object going into one save file is great for viewing, editing, etc.
public class Game
{
private static Game _instance;
private WorldDate _date;
public WorldDate date { get { return _date; } set { _date = value;} }
public string fileName;
public Archmage player { get; set; }
public Character narrator;
public Battle battle;
public int fame { get; set; }
public int turn { get; set; }
public Job currentJob;
public Tower tower { get; set; }
public Dictionary<string, int> expenses { get; set; }
public Queue<GameEvent> events { get; set; }
public List<Location> locations { get; set; }
public List<Town> towns { get; set; }
public List<Noble> nobles { get; set; }
public List<Character> people { get; set; }
public WorldGenerator world;
public SortedDictionary<float, string> feelings;
public Stack<Location> previousLocation { get; set; }
private string logCopy; //for serialization
public Story currentStory { get; set; }Lesson 2: Have a Standard Pattern for Object Initialization
I use shorthand in my C# where I will initialize class variables at time of definition. That’s great until you run into serialization that requires parameterless constructor for recreating your object. None of those shorthand initializations will be called, so your objects will have nulls all over the place.
For example, the old way of:
Public class Character
{
public string Name = “default”;
public Jobtype job;
public List<Relationship> relationships = new List<Relationship>();
}
Means whenever I go to use Name it will already be initialized. Except when it is coming from serialization, then it will be null.
The pattern I settled on (and my 100+ entity objects now all follow) is:
- Public parameterless constructor
- Private Init() method does all initialization
- All constructors call the Init() method.
- Property getter and setter for anything I want serialized.
So the new version looks like this:
Public class Character
{
public string Name;
public Jobtype job {get; set;}
public List<Relationship> relationships {get; set;}
private void Init()
{
Name = “default”;
relationships = new List<Relationship>();
}
public Character()
{
Init();
}
public Character(string _name)
{
Init();
Name = _name;
}
}Lesson 3: A Simple Pattern for Non-Serialized Values
The key to saving data is to only save what you need, not everything you have. If you can recreate an object with a few values then use a function to recreate/recalculate the rest that’s the way to go. One reason is it allows more flexibility for changes later: values recalculated on the class can have their data type and definitions changed at your whim, whereas doing so with a serialized value immediately breaks all previous save games. Definitely not something you want to do after a game ships, even when in early access.
Another example is special objects that you want to save state on which are not part of the super Game object. In my case is I have a game log that tracks everything which happens in the game (for the player, for debugging). It is not on the one super Game objects because I don’t want to write Game.Instance.Logger.log(“…”); all the time. So prior to saving I need to grab a copy of whatever is in the gamelog and post loading I need to copy it back to the gamelog.
So this means there needs to be some special tasks done before/after serialization that the caller of the serialization shouldn’t actually care about. So I made some simple pre/post methods that contain all this specialized code in one simple easy to manage place.
This won’t win any OOP awards, but here is my simple approach to serialization, to serve as a launching pad for your own solution. These methods are on the Game singleton:
public static void Load(string fileName)
{
if (File.Exists(fileName))
{
var serializer = new SharpSerializer();
_instance = serializer.Deserialize(fileName) as Game;
Instance.PostDeserialization();
}
else
{
throw new ArgumentNullException("File not found to load");
}
Logger.log("Game Loaded");
}
public void Save()
{
PrepareForSerialization();
var serializer = new SharpSerializer();
serializer.PropertyProvider.AttributesToIgnore.Clear();
serializer.PropertyProvider.AttributesToIgnore.Add(typeof(XmlIgnoreAttribute));
serializer.Serialize(Instance, fileName);
Logger.log("Game Saved");
}
private void PrepareForSerialization()
{
logCopy = Logger.logText;
}
private void PostDeserialization()
{
Logger.logText = this.logCopy;
}Lesson 4: Ensure You Have Serialization Working Day One
The longer you take to find your perfectly working serialization solution, the more work it will be. I got this one wrong.
As previously mentioned, I did have serialization working for the first 8 months of the project (mostly, I did have the object reference issue). But when it didn’t work in Unity at all, I just continued working without it. This increased Technical Debt.
The number of classes I had to change 10 months later verses day one is probably triple. So having the serialization paradigm right, from the beginning, means every new class is designed correctly up front: no technical debt, no rework.
Lesson 5: Use a Human Readable Format
Probably obvious to many, but whether it is JSON or XML or something else, the ability to see your data in its serialized state is invaluable for debugging. I could tell early on the solution was the right one because of the layout of the data in the file.
Conclusion
Well that is all I have to say about serialization. It works. It will work for this project and any conceivable future projects. I’m happy. Time to move on to much more exciting features.
Let me know in the comments if you have any questions on any of the above.