Canonical Serialization: Why the Same Object Hashes Differently

Two systems hold what they agree is the same object. They serialize it. They hash it. The hashes do not match.

Nothing is broken. That is just how serialization works, and if you built duplicate detection on top of a hash without knowing it, your duplicate detection does not detect duplicates.

Serialization is not a canonical operation

When data moves between systems it gets converted to text, usually JSON these days, sometimes XML, sometimes a binary format. The assumption that quietly gets made is this:

serializeInSystemA(obj) == serializeInSystemB(obj)

It is not true, and there is no reason it should be. Every library makes its own choices. Json.NET writes properties in the order they are declared on the type. Declare the same properties in a different order on the client’s model and you get a different string for the same data. Add whitespace settings, null handling, and numeric formatting, and the surface for divergence gets wider.

For round-tripping, none of this matters. You serialize, you send, the other side deserializes, everybody gets the object they expected. Ordering is irrelevant to a parser.

It starts mattering the moment the serialized form itself becomes meaningful. Hashing a payload to decide whether you have seen this request before is exactly that case. Now the bytes are the identity, and two representations of the same data are two different requests as far as your system is concerned.

The client retries. The properties come across in a different order. Your duplicate check passes. You process the request a second time.

Canonical serialization

Canonical serialization means producing one consistent representation, so that identical data always produces identical output regardless of how it arrived.

The core rule is to sort object properties by name, recursively, all the way down. Nested objects get the same treatment as the root.

Arrays are a design decision, not a rule. If order carries meaning, [A, B, C] and [C, B, A] are genuinely different data, and sorting them destroys the distinction you are trying to detect. If order is incidental, the two are the same request, and leaving them unsorted means you miss the duplicate. There is no safe default here. You have to know your data, per property, and be deliberate. Getting this wrong misrepresents the data in one direction or the other.

Two smaller decisions that bite:

Sort with an ordinal comparer. OrderBy(p => p.Name) uses the current culture’s string comparison. Two servers with different culture settings can order the same property names differently and produce different hashes for identical payloads. Use StringComparer.Ordinal and make the comparison a property of the code rather than a property of the machine.

Decide what nulls mean. Dropping null properties makes {"a":1,"b":null} and {"a":1} hash the same. Sometimes that is right. Sometimes an explicit null is a meaningful instruction and collapsing it hides a real difference. Pick on purpose rather than inheriting whatever NullValueHandling was already set to.

The implementation

Json.NET has no canonical mode, which is the usual reason people reach for a custom ContractResolver. That approach does not work here. A resolver’s CreateProperties runs when Json.NET builds a contract for a typed object by reflection. Parse an arbitrary payload with no target type and you get a JObject, which serializes its tokens back in insertion order and never consults the resolver at all.

Sort the token tree directly instead.

using System;
using System.Linq;
using System.Security.Cryptography;
using System.Text;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

public static class CanonicalPayloadHasher
{
    public static string ComputeHash(string jsonPayload)
    {
        var canonical = Canonicalize(JToken.Parse(jsonPayload));
        var json = canonical.ToString(Formatting.None);

        var bytes = SHA256.HashData(Encoding.UTF8.GetBytes(json));
        return Convert.ToHexString(bytes);
    }

    private static JToken Canonicalize(JToken token)
    {
        switch (token)
        {
            case JObject obj:
                var sorted = new JObject();
                foreach (var property in obj.Properties()
                                            .OrderBy(p => p.Name, StringComparer.Ordinal))
                {
                    sorted.Add(property.Name, Canonicalize(property.Value));
                }
                return sorted;

            case JArray array:
                // Order preserved by default. Where order is not semantically
                // meaningful for a property, sort it here on purpose.
                return new JArray(array.Select(Canonicalize));

            default:
                return token;
        }
    }
}

Worth writing the test that proves it, because the failure is silent:

var a = CanonicalPayloadHasher.ComputeHash("""{"b":1,"a":2}""");
var b = CanonicalPayloadHasher.ComputeHash("""{"a":2,"b":1}""");
// a == b

The same problem and the same fix apply with System.Text.Json. JsonNode preserves insertion order too, and there is no canonical mode there either.

Where this stops being your problem alone

Key ordering is the part that bites first, not the whole surface. 1.0 and 1 are the same number and different text. Unicode escaping, exponent notation, and string normalization all offer more than one way to write the same value. A hand-rolled canonicalizer handles the cases you thought of.

That is tolerable while the hash never leaves your system. It stops being tolerable the moment a client has to compute the same hash you do, which is exactly what happens when you use a payload hash as the operation identifier and expect the caller to rebuild a status URL from it. Your canonicalization rules become published contract, and every edge case you did not think of becomes a support ticket.

RFC 8785 specifies a JSON Canonicalization Scheme for this, covering property ordering and number serialization. If the hash is going over the wire, pointing at a standard is a much better answer than documenting your own.