Entity remapping
Most of the time you don't have any control of the values in Entity
component in the chunk (yes it is an immutable component with 2 integers), it just get assigned as they are created or destroyed.
But ECS library allows you to store Entity
in other components. This is probably to use it as a pointer-like functionality since ECS library provides many API that take Entity
in and you could go anywhere. This pattern is very useful to create relationship between entities.
But it leads to serialization problem. Your setup relationship only works right now in this World
.
If you have an Entity
field hidden somewhere, then that is out of reach of ECS library to accordingly update or invalidate it if the target was destroyed, etc. It is not a real pointer after all.
Imagine I have 5 entities with :
Index Version
1 3
8 4
3 3
4 3
5 3
Each one is allowed to remember 1 Next : IComponentData { Entity next; }
entity, perhaps to implement a linked list -like functionality without pointers. (Then I will "travel" with ComponentDataFromEntity<Next>
, etc.)
Index Version | Next
1 3 5-3
8 4 Entity.Null
3 3 Entity.Null
4 3 Entity.Null
5 3 3-3
If I were to serialize these 5 entities with Next
by hand, what would I do? Naive answer is maybe create an array of Entity[5]
and Next[5]
and store stuff.
But let's think about when we deserialize them. We would have to call EntityManager.CreateEntity
5 times with no way to actually use that Entity[5]
. There is no API to specify the Index
and Version
of the new Entity
you want from the create.
This maybe what we get in a new run, perhaps there are more systems that also create entities that comes before your deserializing system.
Index Version
11 1
12 1
13 2
14 2
15 1
That's right, the new (deserialized, but theoretically, they are new) entities got their 2 integer values according to the rule of the library. You cannot control this.
Next
you can definitely remake based on serialized Next[5]
, but the baked Entity
in it no longer make sense either because newly assigned Entity
rule. The engine do not have any API to forcefully change Entity
of any Entity
. You see it is kind of conflicting itself.
This is what we get in the end. Now I can't relly make use of Next
because it go to somewhere else, most likely the target entity does not exist and they are now just a useless pair of integers.
Index Version | Next
11 1 5-3
12 1 Entity.Null
13 2 Entity.Null
14 2 Entity.Null
15 1 3-3
Don't think about to "deserialize the first thing" to make the Entity
assigned by library more deterministic, to band-aid this problem, it will make the modular design less flexible.
Let's deal with the fact that Entity
's value on creating entities is out of reach. The only way left is to modify the baked Entity
in Next
to comply with the new Entity
assigned by the library. The problem becomes :
- A: Know in the latter situation, which old
Entity
should become what newEntity
. - B: Make a converter, that you feed in one
Entity
and it give out the correctEntity
fast enough. Converter is great because you may have the sameEntity
baked in multiple places. They would give out the same answer fast. - C: Go through every baked
Entity
and use the converter to changeEntity
value. This also sounds like a pain if you have more components likeNext
and you may forgot to update them.
This pattern is called remapping : go check any baked Entity
in IComponentData
and remap them to new value that make the same sense as before serializing. Entity remapping deal with serialization of Entity
value. It make Entity
-as-a-pointer really works 100%. It sounds like you could serialize a pointer when done properly.
It would be used internally by ECS library in net code when synchronizing from server, in serializing API, etc. But here, I will make it simple that we serialize entities by hand. Maybe as simple as storing structure of array (SOA) of each components in ScriptableObject
.
Knowing which entity should become which
Index Version | Next
1 3 5-3
8 4 Entity.Null
3 3 Entity.Null
4 3 Entity.Null
5 3 3-3
Index Version | Next
11 1 5-3
12 1 Entity.Null
13 2 Entity.Null
14 2 Entity.Null
15 1 3-3
From this before-after, instead of serializing just Entity[5]
(which is useless, as explained earlier) you want to also serialize some kind of bridge that help change any baked Entity
to be up to date with the real one. Your own identification that is in your control.
The simplest would be like making a new component MyId
:
Index Version | Next MyId
1 3 5-3 1
8 4 Entity.Null 2
3 3 Entity.Null 3
4 3 Entity.Null 4
5 3 3-3 5
Then later you deserialize, you get that back :
Index Version | Next MyId
11 1 5-3 1
12 1 Entity.Null 2
13 2 Entity.Null 3
14 2 Entity.Null 4
15 1 3-3 5
So imagine keeping a NativeArray<Entity>
serialized along with everything else. Make sure MyId
runs completely in sequence and has no holes :
[1-3][8-4][3-3][4-3][5-3]
Having that serialized, you can make a converter out of the latter situation.
Go through deserialized MyId
component of each entity, the number should be the same. Use the integer in MyId
to index into your serialized NativeArray<Entity>
. and check the current Entity
assigned by the library.
You have got your mapping for the converter.
1-3 -> 11-1
8-4 -> 12-1
3-3 -> 13-2
4-3 -> 14-2
5-3 -> 15-1
There maybe other ways than MyId
approach I use to keep the link to the previous assigned Entity
, it is up to you. But from now I assume you know all correct answer to turn which Entity
to what Entity
.
What's left is just go through Next
component and similar components with baked Entity
and update them. So we need to turn that conversion into a portable converter that is fast enough and ideally can be thrown into jobs.
You can replace MyId
component with something like PreviousEntity : IComponentData { Entity previousEntity; }
. Then you basically get all the Entity
to Entity
answers as soon as you deserialize back. Simple! However keep in mind you need 8 bytes to serialize the previous Entity
. With MyId
, it is possible that you use byte
or ushort
instead to save serialize space. ushort
could account for about 60000 entities for 2 bytes. Plus, you may use MyId
for other purposes. Maybe each of your thing need a sequential identification anyways.
In ECS lib, there is an official name of this Entity
to Entity
converter...
NativeArray<EntityRemapInfo>
This is your "old to new Entity
converter". It is an ample memory area that allows fast remap, that is, it could change one Entity
to the correct Entity
. I would call this NAERI from now. We are not talking about how to get this yet, imagine we have one already. And also remember we know all the correct Entity
to Entity
answer from the previous section. We just need to make this converter.
You may think of something like NativeHashMap<Entity,Entity>
when remapping so you can turn one into another in O(1)
time with the power of hashing (2 integers).
But by sacrificing some memory you can do faster than that. Recall the previous situation :
Index Version
1 3
8 4
3 3
4 3
5 3
Index Version | Next
11 1 5-3
12 1 Entity.Null
13 2 Entity.Null
14 2 Entity.Null
15 1 3-3
You would like something that when you feed in 5-3
it instantly becomes 15-1
, and if you feed 3-3
it becomes 13-2
. So the result looks like this :
Index Version | Next
11 1 15-1
12 1 Entity.Null
13 2 Entity.Null
14 2 Entity.Null
15 1 13-2
The NativeHashMap<Entity,Entity>
could work. e.g. myNhm[5-3]
then it should give out 15-1
somehow.
But consider this plain oversized (length 8) NativeArray
:
Index Data in it
1
2
3 3, 13-2
4
5 3, 15-1
6
7
8
If you have this thing, then you can just go through all your baked Entity
in Next
, and use just the Index
part to index into the NAERI (that's why it has to be oversized and cover the highest index number), then if
with the remaining version number part stored in that index (the index already verified the index part), if passed, give out the stored answer. You need to be convinced first that this is better performance wise than using Entity
as a key to NativeHashMap<Entity,Entity>
.
The "data in it" you see is this EntityRemapInfo
.
public struct EntityRemapInfo
{
public int SourceVersion;
public Entity Target;
}
If you are still confused, think what a converter is. It would be a pair of Entity
(so 4 integers, 2 for source, 2 for target). But this struct
has 3 integers, because the index part of the source has been cleverly verified by the indexing into the array instead.
This improves performance at the cost of memory because you no longer have to hash. Imagine if you had more than 1,000,000 entities and you just want to convert the previous high indexed but low versioned 998,114-1
Entity
to whatever it is now. You need NAERI sized 1,000,000 just to "efficiently" convert this one entity. (such array would cost 4*4*1000000 = 16 MB) If the conversion is sparse, maybe NativeHashMap
with few Entity
to Entity
entries maybe better.
Also, NativeArray
could be thrown into a job. Imagine remapping different components in parallel really fast by multiple jobs.
To make a NAERI (providing you know all the Entity
to Entity
pairs already, which you know from MyId
approach from the previous section), you can use some utilities in EntityRemapUtility
.
SparseEntityRemapInfo
public struct SparseEntityRemapInfo
{
public Entity Src;
public Entity Target;
}
The talk about EntityRemapInfo
's 3 integers approach wasting space, so they know you would complain so they also made a version that more simply contains a pair of Entity
. Straight to the point. But it would be less efficient to convert with this but no longer waste space when you have high number of Index
. It is used in some API but this is all you should know about it now. Let's get back to NAERI. (This one would be NASERI, the abbreviation starts to get cringier somehow.)
EntityRemapUtility
This is a static
class containing tools so we could remap by hand. Library methods like EntityManager.MoveEntitiesFrom
would utilize things in here. To understand these tools, let's use them manually.
This is a huge list of exposed public
methods, with no documentation currently.
public static void GetTargets(out NativeArray<Entity> output, NativeArray<EntityRemapInfo> remapping)
public static void AddEntityRemapping(ref NativeArray<EntityRemapInfo> remapping, Entity source, Entity target)
public static Entity RemapEntity(ref NativeArray<EntityRemapInfo> remapping, Entity source)
public static Entity RemapEntity(EntityRemapInfo* remapping, Entity source)
public static Entity RemapEntityForPrefab(SparseEntityRemapInfo* remapping, int remappingCount, Entity source)
public static EntityOffsetInfo[] CalculateEntityOffsets<T>()
public static bool HasEntityMembers(Type type)
public static EntityOffsetInfo[] CalculateEntityOffsets(Type type)
public static EntityPatchInfo* AppendEntityPatches(EntityPatchInfo* patches, EntityOffsetInfo* offsets, int offsetCount, int baseOffset, int stride)
public static BufferEntityPatchInfo* AppendBufferEntityPatches(BufferEntityPatchInfo* patches, EntityOffsetInfo* offsets, int offsetCount, int bufferBaseOffset, int bufferStride, int elementStride)
public static ManagedEntityPatchInfo* AppendManagedEntityPatches(ManagedEntityPatchInfo* patches, ComponentType type)
public static void PatchEntities(EntityOffsetInfo[] scalarPatches, byte* chunkBuffer, ref NativeArray<EntityRemapInfo> remapping)
public static void PatchEntities(EntityPatchInfo* scalarPatches, int scalarPatchCount,BufferEntityPatchInfo* bufferPatches, int bufferPatchCount,byte* chunkBuffer, int entityCount, ref NativeArray<EntityRemapInfo> remapping)
public static void PatchEntitiesForPrefab(EntityPatchInfo* scalarPatches, int scalarPatchCount, BufferEntityPatchInfo* bufferPatches, int bufferPatchCount, byte* chunkBuffer, int indexInChunk, int entityCount, SparseEntityRemapInfo* remapping, int remappingCount)
Making the converter
We are ready to demystify EntityRemapUtility
class one by one.
So you can just new NativeArray<EntityRemapInfo>(8, Allocator.Persistent)
(8 because in the example situation, I know the most Entity
index goes up to 8) so it start out empty first.
Then with all pairs of Entity
from-to we know from the MyId
approach, for each pair of answer, put your empty NAERI into that ref
:
public static void AddEntityRemapping(ref NativeArray<EntityRemapInfo> remapping, Entity source, Entity target)
It would help you completing the NAERI. Remember that the size need to be large enough cover the largest Index
part of the source Entity
. So, when you serialize along with MyId
you should scout for the largest Index
part in the old Entity
and also serialize that number so you can make correctly sized converter on deserialize without linear search.
*If you found that your largest Entity.Index
to remap is 428, then at new NativeArray<EntityRemapInfo>(__, Allocator.__)
construction you need to give it a size 429. This is so that the maximum indexable value is 428.
Beware, if you are serializing 300 entities it is not necessary that you need 300 sized NAERI. The Index
part may have went much higher than that and with NAERI's approach we unfortunately need the index for mapping in need for speed.
Additionally let's talk a bit about this utility method :
public static void GetTargets(out NativeArray<Entity> output, NativeArray<EntityRemapInfo> remapping)
Basically it just get the target part (2 integers out of 3 integers of EntityRemapInfo
) out of your NAERI. Which is kinda useless in our remapping task as we don't really want to know that, we want to update all the baked entities. But it is used in EntityManager.MoveEntitiesFrom
where it would give you back the move result.
Ready to remap
You are ready to use your NAERI, finally? Remembered that how to actually use it is to index into it with Index
part, then if
for Version
part, then grab the answer.
But! Instead of doing that you could use this helper in EntityRemapUtility
so you don't reinvent the wheel :
public static Entity RemapEntity(ref NativeArray<EntityRemapInfo> remapping, Entity source)
This properly returns Entity.Null
in the case of no target available in the NAERI.
Note that you can use static
methods in jobs. It would simply be a part of Burst compiled assembly for each job. So imagine you are remapping things in Entities.ForEach( (ref Next next) => ...
then you throw in your NAERI. You use that static
method in here. Because ref
, you can write back the returned Entity
to the deserialized Next
with wrong Entity
to correct one. Done!
Bonus : Need more magic?
The previous usage of NAERI is possible because you know your Next
contains something to be fixed, then you make a system that schedule Entities.ForEach( (ref Next next) => ...
kind of parallel job to fix them.
Is there anything more magical than this? Like scan for all types that needs fixing, then fix them all. There is a limit of how magical the library could know about your game, but there are some.
public static bool HasEntityMembers(Type type)
In the case that you forgot if you have Entity
baked in Next
, throwing Next
type in here would return true
. It might sounds stupid but it is a tool for more automated remapping. Maybe you have a IEnumerable<Type>
of serialized types. You can simply iterate through them and use this utility to check if there are anything worth remapping nested (recursively or not) inside.
Then how would you patch the said type without writing a job to reassign Entity
?
public static EntityOffsetInfo[] CalculateEntityOffsets(Type type)
public static void PatchEntities(EntityOffsetInfo[] scalarPatches, byte* chunkBuffer, ref NativeArray<EntityRemapInfo> remapping)
It looks like hell but it seems to be able to magically patch everything with NAERI and some "info" from the type you wish to remap, to any single chunk.
That method has byte*
chunk buffer. So if you want to use them you need to go unsafe
first. Also I don't know how would you get a hold of chunk memory directly, because Chunk
struct is internal
. And they are not documented and used by only EntityManager.MoveEntitiesFrom
. It make sense that this method would try to magically make baked Entity
OK after the movement.
The only way I could think of is to understand the built-in serialization API ( SerializeUtility
) and hack out the serialized buffer, then grab the chunk part (how?), then feed to this method. This should be enough that I think you should NOT mess with them. Rather, the serialization API should have done this for us. I think it may already be the case that you can "just" serialize any component with baked Entity
and get back with them remapped to make sense. But the point of this article is to remap by hand if you don't want to use the serialization API. For example, I would like to serialize entities to Protobuf and I don't want to serialize just chunks memories. They are too hardcore and prone to lose forward compatibility.
Just schedule a job manually to remap with NAERI and dig out all your Entity
fields on your own.