Entity remapping

Most of the time you don't have any control of the values in Entity component in the chunk (yes it is an immutable component with 2 integers), it just get assigned as they are created or destroyed.

But ECS library allows you to store Entity in other components. This is probably to use it as a pointer-like functionality since ECS library provides many API that take Entity in and you could go anywhere. This pattern is very useful to create relationship between entities.

But it leads to serialization problem. Your setup relationship only works right now in this World.

If you have an Entity field hidden somewhere, then that is out of reach of ECS library to accordingly update or invalidate it if the target was destroyed, etc. It is not a real pointer after all.

Imagine I have 5 entities with :

Index Version
1       3
8       4
3       3
4       3
5       3

Each one is allowed to remember 1 Next : IComponentData { Entity next; } entity, perhaps to implement a linked list -like functionality without pointers. (Then I will "travel" with ComponentDataFromEntity<Next>, etc.)

Index Version | Next
1       3        5-3
8       4        Entity.Null
3       3        Entity.Null
4       3        Entity.Null
5       3        3-3

If I were to serialize these 5 entities with Next by hand, what would I do? Naive answer is maybe create an array of Entity[5] and Next[5] and store stuff.

But let's think about when we deserialize them. We would have to call EntityManager.CreateEntity 5 times with no way to actually use that Entity[5]. There is no API to specify the Index and Version of the new Entity you want from the create.

This maybe what we get in a new run, perhaps there are more systems that also create entities that comes before your deserializing system.

Index Version
11       1
12       1
13       2
14       2
15       1

That's right, the new (deserialized, but theoretically, they are new) entities got their 2 integer values according to the rule of the library. You cannot control this.

Next you can definitely remake based on serialized Next[5], but the baked Entity in it no longer make sense either because newly assigned Entity rule. The engine do not have any API to forcefully change Entity of any Entity. You see it is kind of conflicting itself.

This is what we get in the end. Now I can't relly make use of Next because it go to somewhere else, most likely the target entity does not exist and they are now just a useless pair of integers.

Index Version | Next
11       1       5-3
12       1       Entity.Null
13       2       Entity.Null
14       2       Entity.Null
15       1       3-3

Don't think about to "deserialize the first thing" to make the Entity assigned by library more deterministic, to band-aid this problem, it will make the modular design less flexible.

Let's deal with the fact that Entity's value on creating entities is out of reach. The only way left is to modify the baked Entity in Next to comply with the new Entity assigned by the library. The problem becomes :

• A: Know in the latter situation, which old Entity should become what new Entity.
• B: Make a converter, that you feed in one Entity and it give out the correct Entity fast enough. Converter is great because you may have the same Entity baked in multiple places. They would give out the same answer fast.
• C: Go through every baked Entity and use the converter to change Entity value. This also sounds like a pain if you have more components like Next and you may forgot to update them.

This pattern is called remapping : go check any baked Entity in IComponentData and remap them to new value that make the same sense as before serializing. Entity remapping deal with serialization of Entity value. It make Entity-as-a-pointer really works 100%. It sounds like you could serialize a pointer when done properly.

It would be used internally by ECS library in net code when synchronizing from server, in serializing API, etc. But here, I will make it simple that we serialize entities by hand. Maybe as simple as storing structure of array (SOA) of each components in ScriptableObject.

Knowing which entity should become which

Index Version | Next
1       3        5-3
8       4        Entity.Null
3       3        Entity.Null
4       3        Entity.Null
5       3        3-3

Index Version | Next
11       1       5-3
12       1       Entity.Null
13       2       Entity.Null
14       2       Entity.Null
15       1       3-3

From this before-after, instead of serializing just Entity[5] (which is useless, as explained earlier) you want to also serialize some kind of bridge that help change any baked Entity to be up to date with the real one. Your own identification that is in your control.

The simplest would be like making a new component MyId :

Index Version | Next             MyId
1       3        5-3              1
8       4        Entity.Null      2
3       3        Entity.Null      3
4       3        Entity.Null      4
5       3        3-3              5

Then later you deserialize, you get that back :

Index Version | Next            MyId
11       1       5-3             1
12       1       Entity.Null     2
13       2       Entity.Null     3
14       2       Entity.Null     4
15       1       3-3             5

So imagine keeping a NativeArray<Entity> serialized along with everything else. Make sure MyId runs completely in sequence and has no holes :

[1-3][8-4][3-3][4-3][5-3]

Having that serialized, you can make a converter out of the latter situation.

Go through deserialized MyId component of each entity, the number should be the same. Use the integer in MyId to index into your serialized NativeArray<Entity>. and check the current Entity assigned by the library.

You have got your mapping for the converter.

1-3 -> 11-1
8-4 -> 12-1
3-3 -> 13-2
4-3 -> 14-2
5-3 -> 15-1

There maybe other ways than MyId approach I use to keep the link to the previous assigned Entity, it is up to you. But from now I assume you know all correct answer to turn which Entity to what Entity.

What's left is just go through Next component and similar components with baked Entity and update them. So we need to turn that conversion into a portable converter that is fast enough and ideally can be thrown into jobs.

You can replace MyId component with something like PreviousEntity : IComponentData { Entity previousEntity; }. Then you basically get all the Entity to Entity answers as soon as you deserialize back. Simple! However keep in mind you need 8 bytes to serialize the previous Entity. With MyId, it is possible that you use byte or ushort instead to save serialize space. ushort could account for about 60000 entities for 2 bytes. Plus, you may use MyId for other purposes. Maybe each of your thing need a sequential identification anyways.

In ECS lib, there is an official name of this Entity to Entity converter...

NativeArray<EntityRemapInfo>

This is your "old to new Entity converter". It is an ample memory area that allows fast remap, that is, it could change one Entity to the correct Entity. I would call this NAERI from now. We are not talking about how to get this yet, imagine we have one already. And also remember we know all the correct Entity to Entity answer from the previous section. We just need to make this converter.

You may think of something like NativeHashMap<Entity,Entity> when remapping so you can turn one into another in O(1) time with the power of hashing (2 integers).

But by sacrificing some memory you can do faster than that. Recall the previous situation :

Index Version
1       3
8       4
3       3
4       3
5       3

Index Version | Next
11       1       5-3
12       1       Entity.Null
13       2       Entity.Null
14       2       Entity.Null
15       1       3-3

You would like something that when you feed in 5-3 it instantly becomes 15-1, and if you feed 3-3 it becomes 13-2. So the result looks like this :

Index Version | Next
11       1       15-1
12       1       Entity.Null
13       2       Entity.Null
14       2       Entity.Null
15       1       13-2

The NativeHashMap<Entity,Entity> could work. e.g. myNhm[5-3] then it should give out 15-1 somehow.

But consider this plain oversized (length 8) NativeArray :

Index   Data in it
1        
2
3         3, 13-2
4         
5         3, 15-1
6
7
8        

If you have this thing, then you can just go through all your baked Entity in Next, and use just the Index part to index into the NAERI (that's why it has to be oversized and cover the highest index number), then if with the remaining version number part stored in that index (the index already verified the index part), if passed, give out the stored answer. You need to be convinced first that this is better performance wise than using Entity as a key to NativeHashMap<Entity,Entity>.

The "data in it" you see is this EntityRemapInfo.

public struct EntityRemapInfo
{
    public int SourceVersion;
    public Entity Target;
}

If you are still confused, think what a converter is. It would be a pair of Entity (so 4 integers, 2 for source, 2 for target). But this struct has 3 integers, because the index part of the source has been cleverly verified by the indexing into the array instead.

This improves performance at the cost of memory because you no longer have to hash. Imagine if you had more than 1,000,000 entities and you just want to convert the previous high indexed but low versioned 998,114-1 Entity to whatever it is now. You need NAERI sized 1,000,000 just to "efficiently" convert this one entity. (such array would cost 4*4*1000000 = 16 MB) If the conversion is sparse, maybe NativeHashMap with few Entity to Entity entries maybe better.

Also, NativeArray could be thrown into a job. Imagine remapping different components in parallel really fast by multiple jobs.

To make a NAERI (providing you know all the Entity to Entity pairs already, which you know from MyId approach from the previous section), you can use some utilities in EntityRemapUtility.

SparseEntityRemapInfo

public struct SparseEntityRemapInfo
{
    public Entity Src;
    public Entity Target;
}

The talk about EntityRemapInfo's 3 integers approach wasting space, so they know you would complain so they also made a version that more simply contains a pair of Entity. Straight to the point. But it would be less efficient to convert with this but no longer waste space when you have high number of Index. It is used in some API but this is all you should know about it now. Let's get back to NAERI. (This one would be NASERI, the abbreviation starts to get cringier somehow.)

EntityRemapUtility

This is a static class containing tools so we could remap by hand. Library methods like EntityManager.MoveEntitiesFrom would utilize things in here. To understand these tools, let's use them manually.

This is a huge list of exposed public methods, with no documentation currently.

public static void GetTargets(out NativeArray<Entity> output, NativeArray<EntityRemapInfo> remapping)
public static void AddEntityRemapping(ref NativeArray<EntityRemapInfo> remapping, Entity source, Entity target)
public static Entity RemapEntity(ref NativeArray<EntityRemapInfo> remapping, Entity source)
public static Entity RemapEntity(EntityRemapInfo* remapping, Entity source)
public static Entity RemapEntityForPrefab(SparseEntityRemapInfo* remapping, int remappingCount, Entity source)
public static EntityOffsetInfo[] CalculateEntityOffsets<T>()
public static bool HasEntityMembers(Type type)
public static EntityOffsetInfo[] CalculateEntityOffsets(Type type)
public static EntityPatchInfo* AppendEntityPatches(EntityPatchInfo* patches, EntityOffsetInfo* offsets, int offsetCount, int baseOffset, int stride)
public static BufferEntityPatchInfo* AppendBufferEntityPatches(BufferEntityPatchInfo* patches, EntityOffsetInfo* offsets, int offsetCount, int bufferBaseOffset, int bufferStride, int elementStride)
public static ManagedEntityPatchInfo* AppendManagedEntityPatches(ManagedEntityPatchInfo* patches, ComponentType type)
public static void PatchEntities(EntityOffsetInfo[] scalarPatches, byte* chunkBuffer, ref NativeArray<EntityRemapInfo> remapping)
public static void PatchEntities(EntityPatchInfo* scalarPatches, int scalarPatchCount,BufferEntityPatchInfo* bufferPatches, int bufferPatchCount,byte* chunkBuffer, int entityCount, ref NativeArray<EntityRemapInfo> remapping)
public static void PatchEntitiesForPrefab(EntityPatchInfo* scalarPatches, int scalarPatchCount, BufferEntityPatchInfo* bufferPatches, int bufferPatchCount, byte* chunkBuffer, int indexInChunk, int entityCount, SparseEntityRemapInfo* remapping, int remappingCount)

Making the converter

We are ready to demystify EntityRemapUtility class one by one.

So you can just new NativeArray<EntityRemapInfo>(8, Allocator.Persistent) (8 because in the example situation, I know the most Entity index goes up to 8) so it start out empty first.

Then with all pairs of Entity from-to we know from the MyId approach, for each pair of answer, put your empty NAERI into that ref :

public static void AddEntityRemapping(ref NativeArray<EntityRemapInfo> remapping, Entity source, Entity target)

It would help you completing the NAERI. Remember that the size need to be large enough cover the largest Index part of the source Entity. So, when you serialize along with MyId you should scout for the largest Index part in the old Entity and also serialize that number so you can make correctly sized converter on deserialize without linear search.

*If you found that your largest Entity.Index to remap is 428, then at new NativeArray<EntityRemapInfo>(__, Allocator.__) construction you need to give it a size 429. This is so that the maximum indexable value is 428.

Beware, if you are serializing 300 entities it is not necessary that you need 300 sized NAERI. The Index part may have went much higher than that and with NAERI's approach we unfortunately need the index for mapping in need for speed.

Additionally let's talk a bit about this utility method :

public static void GetTargets(out NativeArray<Entity> output, NativeArray<EntityRemapInfo> remapping)

Basically it just get the target part (2 integers out of 3 integers of EntityRemapInfo)  out of your NAERI. Which is kinda useless in our remapping task as we don't really want to know that, we want to update all the baked entities. But it is used in EntityManager.MoveEntitiesFrom where it would give you back the move result.

Ready to remap

You are ready to use your NAERI, finally? Remembered that how to actually use it is to index into it with Index part, then if for Version part, then grab the answer.

But! Instead of doing that you could use this helper in EntityRemapUtility so you don't reinvent the wheel :

public static Entity RemapEntity(ref NativeArray<EntityRemapInfo> remapping, Entity source)

This properly returns Entity.Null in the case of no target available in the NAERI.

Note that you can use static methods in jobs. It would simply be a part of Burst compiled assembly for each job. So imagine you are remapping things in Entities.ForEach( (ref Next next) => ... then you throw in your NAERI. You use that static method in here. Because ref, you can write back the returned Entity to the deserialized Next with wrong Entity to correct one. Done!

Bonus : Need more magic?

The previous usage of NAERI is possible because you know your Next contains something to be fixed, then you make a system that schedule Entities.ForEach( (ref Next next) => ... kind of parallel job to fix them.

Is there anything more magical than this? Like scan for all types that needs fixing, then fix them all. There is a limit of how magical the library could know about your game, but there are some.

public static bool HasEntityMembers(Type type)

In the case that you forgot if you have Entity baked in Next, throwing Next type in here would return true. It might sounds stupid but it is a tool for more automated remapping. Maybe you have a IEnumerable<Type> of serialized types. You can simply iterate through them and use this utility to check if there are anything worth remapping nested (recursively or not) inside.

Then how would you patch the said type without writing a job to reassign Entity?

public static EntityOffsetInfo[] CalculateEntityOffsets(Type type)

public static void PatchEntities(EntityOffsetInfo[] scalarPatches, byte* chunkBuffer, ref NativeArray<EntityRemapInfo> remapping)

It looks like hell but it seems to be able to magically patch everything with NAERI and some "info" from the type you wish to remap, to any single chunk.

That method has byte* chunk buffer. So if you want to use them you need to go unsafe first. Also I don't know how would you get a hold of chunk memory directly, because Chunk struct is internal. And they are not documented and used by only EntityManager.MoveEntitiesFrom. It make sense that this method would try to magically make baked Entity OK after the movement.

The only way I could think of is to understand the built-in serialization API ( SerializeUtility ) and hack out the serialized buffer, then grab the chunk part (how?), then feed to this method. This should be enough that I think you should NOT mess with them. Rather, the serialization API should have done this for us. I think it may already be the case that you can "just" serialize any component with baked Entity and get back with them remapped to make sense. But the point of this article is to remap by hand if you don't want to use the serialization API. For example, I would like to serialize entities to Protobuf and I don't want to serialize just chunks memories. They are too hardcore and prone to lose forward compatibility.

Just schedule a job manually to remap with NAERI and dig out all your Entity fields on your own.