Memory consumption

[piotrkowalczuk]

Hello Team,

I am raising this issue to get an idea of whether we should choose protovalidate-go or protoc-gen-validate. As reported in other issues, some performance penalty is expected:

• Increased memory consumption in v0.9.x #190
• [Question] Performance issues (compared with PGV) #164

I am wondering what the target performance is. I ran a couple of benchmarks to estimate the effective memory utilization increase compared to our baseline. I have three scenarios:

• Message encoding: 370MB
• Message encoding + validation: 5800MB ❗
• Message encoding + optimized validation: 5500MB ❗ (CEL alone accounts for 3600MB)

The payload used for measuring has approximately 50 fields, nested in different configurations.

I'm wondering what we can do on our end to adopt the library:

• Are the reported values within the range of expected utilization?
• Is there a way to disable CEL, for example, if the directive is never used?
• anything on the roadmap to take the resource consumption down?

func BenchmarkMarshalAndValidate(b *testing.B) {
	cases := map[string]proto.Message{
		"medium": createSampleProtoMessage(),
		"large":  createLargeProtoMessage(),
	}

	for hint, msg := range cases {
		b.Run(hint, func(b *testing.B) {
			validator, err := protovalidate.New()
			if err != nil {
				b.Fatal(err)
			}

			b.ReportAllocs()
			b.ResetTimer()

			for n := 0; n < b.N+N; n++ {
				if err := validator.Validate(msg); err != nil {
					b.Fatal(err)
				}

				_, err := proto.Marshal(msg)
				if err != nil {
					b.Fatal(err)
				}
			}
		})
	}
}

func BenchmarkOptimizedValidateAndMarshal(b *testing.B) {
	cases := map[string]proto.Message{
		"medium": createSampleProtoMessage(),
		"large":  createLargeProtoMessage(),
	}

	for hint, msg := range cases {
		b.Run(hint, func(b *testing.B) {
			val, err := protovalidate.New(
				protovalidate.WithMessages(&schemav1.IdentifierEventOrder{}),
				//protovalidate.WithDisableLazy(),
				protovalidate.WithFailFast(),
			)
			if err != nil {
				b.Fatal(err)
			}
			if err := val.Validate(msg); err != nil {
				b.Fatal(err)
			}

			b.ReportAllocs()
			b.ResetTimer()

			for n := 0; n < b.N+N; n++ {
				if err := val.Validate(msg); err != nil {
					b.Fatal(err)
				}

				_, err := proto.Marshal(msg)
				if err != nil {
					b.Fatal(err)
				}
			}
		})
	}
}

cpu: Apple M1 Pro
BenchmarkJustMarshal/medium-10         	 2846248	       375.1 ns/op	     182 B/op	       1 allocs/op
BenchmarkJustMarshal/large-10          	 1490960	       803.4 ns/op	     409 B/op	       1 allocs/op
BenchmarkMarshalAndValidate/medium-10         	   94866	     10813 ns/op	    2245 B/op	      84 allocs/op
BenchmarkMarshalAndValidate/large-10          	   55632	     18842 ns/op	    4192 B/op	     138 allocs/op
BenchmarkOptimizedValidateAndMarshal/medium-10         	  108760	     10050 ns/op	    1996 B/op	      78 allocs/op
BenchmarkOptimizedValidateAndMarshal/large-10          	   54223	     18850 ns/op	    4074 B/op	     139 allocs/op

[rodaine]

Hi, @piotrkowalczuk! Thanks for the detailed report.

Let me get your questions out of the way before providing a more holistic response:

Are the reported values within the range of expected utilization?

They are expected, but still not ideal, particularly for large and complex messages.

For simpler messages, we expect sub-microsecond validation that scales ~linearly with the number of rules applied (modulo repeated/map fields and linear CEL expressions), and our goal is to minimize allocations. Recent changes to support better violation errors have resulted in extra memory allocations, and there's been an outstanding limitation around map fields that make them perform sub-optimally (both in memory and cpu).

All in all, we're continually looking to improve performance here.

Is there a way to disable CEL, for example, if the directive is never used?

Every rule in protovalidate is a CEL expression (with only a couple of exceptions), including the standard ones. The CEL work you're seeing in the profile diagram is after we've optimized the actual CEL expressions maximally and cached them for each message passed to the validator. The large NewHierarchicalActivation is essentially the variables being created for expressions; this should not be this dominating and requires more investigation.

anything on the roadmap to take the resource consumption down?

Yep! Now that we've hit v1, the plan is to continue iterating on the internals to squeeze out better performance. We also look to the community if there's any interest in helping with that endeavor.

---

We have a KitchenSinkMessage we use in conformance for a holistic check on behavior, and use some rather small messages for baseline benchmarks, but we'd love more details on the shape of the messages you were benchmarking with. If you could provide them (redacted as needed), that would be helpful for us to dig in.

One final note: protoc-gen-validate is in maintenance mode.

Let us know if you have further questions!

[piotrkowalczuk]

Hi @rodaine , thanks for swift response! The message wire wise looks more or less like this:

message MessageA {
  MessageB field_1 = 1;
  optional MessageB field_2 = 2;

  oneof payload_oneof {
    MessageC field_50 = 50;
    MessageD field_51 = 51;
    MessageE field_52 = 52;
    MessageF field_53 = 53;
    MessageG field_54 = 54;
    MessageH field_55 = 55;
    MessageI field_56 = 56;
    MessageJ field_57 = 57;
    MessageK field_58 = 58;
    MessageL field_59 = 59;
    MessageM field_60 = 60;
    MessageN field_61 = 61;
    MessageO field_62 = 62;
    MessageP field_63 = 63;
  }
}

message MessageC {
  string field_1 = 1;
  string field_2 = 2;
  string field_3 = 3;
  optional string field_4 = 4;
  optional string field_5 = 5;
  optional string field_6 = 6;
  optional MessageQ field_7 = 7;
  optional MessageQ field_8 = 8;
  MessageR field_9 = 9;

  oneof X_oneof {
    MessageS field_10 = 10;
    MessageT field_11 = 11;
    MessageU field_12 = 12;
    MessageV field_13 = 13;
    MessageW field_30 = 30;
    MessageX field_31 = 31;
    MessageY field_32 = 32;
    MessageZ field_37 = 37;
  }

  optional string field_14 = 14;
  MessageAA field_15 = 15;

  oneof XX_oneof {
    MessageAB field_16 = 16;
    MessageAC field_17 = 17;
    MessageAD field_18 = 18;
    MessageAE field_19 = 19;
    MessageAF field_33 = 33;
  }

  optional MessageAG field_20 = 20;
  repeated MessageAH field_21 = 21;
  optional string field_24 = 24;
  optional string field_25 = 25;
  optional string field_26 = 26;
  optional MessageAI field_29 = 29;
  optional string field_35 = 35;
  bool field_36 = 36;

  message MessageAI {
    int32 field_1 = 1;
    int32 field_2 = 2;
  }

  message MessageAG {
    optional string field_1 = 1;
    optional string field_2 = 2;
    optional string field_3 = 3;
    optional string field_4 = 4;
    optional string field_5 = 5;
    optional string field_6 = 6;
    optional string field_7 = 7;
    optional string field_8 = 8;
    optional string field_9 = 9;
    optional string field_10 = 10;
    bool field_11 = 11;
    optional string field_12 = 12;
    optional string field_13 = 13;
    optional string field_14 = 14;
    optional string field_15 = 15;
    optional string field_16 = 16;
  }

  message MessageR {
    optional int64 field_1 = 1;
    optional int64 field_2 = 2;
    string field_3 = 3;
  }

  message MessageAA {
    optional string field_1 = 1;
    optional string field_2 = 2;
    optional string field_3 = 3;
  }

  message MessageAH {
    optional int32 field_1 = 1;
    optional int32 field_2 = 2;
    optional int32 field_3 = 3;
    optional int32 field_4 = 4;
    optional int32 field_5 = 5;
    repeated MessageAJ field_6 = 6;
    optional string field_7 = 7;
  }

  message MessageAJ {
    string field_1 = 1;
    string field_2 = 2;
    int32 field_3 = 3;
    optional int32 field_4 = 4;
    optional int32 field_11 = 11;
    optional int32 field_5 = 5;
    optional int32 field_6 = 6;
    optional string field_7 = 7;
    optional string field_8 = 8;
    optional string field_9 = 9;
    optional string field_10 = 10;
  }

  message MessageS {}

  message MessageT {}

  message MessageU {
    string field_1 = 1;
    string field_2 = 2;
    optional string field_3 = 3;
  }

  message MessageV {}

  message MessageW {}

  message MessageX {}

  message MessageY {}

  message MessageZ {}

  message MessageAB {
    optional int32 field_1 = 1;
  }

  message MessageAD {}

  message MessageAC {
    optional int32 field_1 = 1;
  }

  message MessageAE {
    string field_1 = 1;
  }

  message MessageAF {}
}

message MessageD {
  string field_1 = 1;
  string field_2 = 2;
  string field_3 = 3;
  optional string field_4 = 4;
  optional string field_5 = 5;
  optional string field_6 = 6;
}

message MessageE {
  string field_1 = 1;
  string field_2 = 2;
}

message MessageF {
  string field_1 = 1;
  string field_2 = 2;
}

message MessageG {
  string field_1 = 1;
  string field_2 = 2;
  optional string field_3 = 3;
  optional string field_4 = 4;
}

message MessageH {
  string field_1 = 1;
  optional string field_2 = 2;
  optional string field_3 = 3;
  optional string field_4 = 4;
  bool field_5 = 5;
}

message MessageI {
  string field_1 = 1;
  string field_2 = 2;
  optional bool field_3 = 3;
}

message MessageJ {
  string field_1 = 1;
  optional string field_2 = 2;
  optional MessageAK field_3 = 3;
  repeated MessageAL field_4 = 4;
  optional MessageQ field_5 = 5;
  optional int32 field_6 = 6;
  optional string field_7 = 7;
  optional string field_8 = 8;
  bool field_9 = 9;

  message MessageAK {
    optional string field_1 = 1;
    optional string field_2 = 2;
    optional string field_3 = 3;
    optional string field_4 = 4;
    optional string field_9 = 9;
    optional string field_5 = 5;
    optional string field_6 = 6;
    optional string field_7 = 7;
    optional string field_8 = 8;
    optional string field_10 = 10;
    optional string field_11 = 11;
    optional string field_12 = 12;
    optional string field_13 = 13;
    optional string field_14 = 14;
  }

  message MessageAL {
    optional string field_1 = 1;
    optional string field_2 = 2;
    optional string field_3 = 3;
    optional bool field_4 = 4;
  }
}

message MessageK {
  string field_1 = 1;
  optional MessageQ field_2 = 2;
  optional string field_3 = 3;
  optional MessageAM field_4 = 4;
  string field_5 = 5;
  optional string field_6 = 6;
  optional string field_7 = 7;
  optional string field_8 = 8;
  repeated MessageAN field_9 = 9;
  oneof payment_method_oneof {
    option (entitystate.oneof_options).skip_prefix = true;

    MessageAO field_10 = 10;
    MessageAP field_11 = 11;
    MessageAQ field_12 = 12;
    MessageAR field_13 = 13;
    MessageAS field_14 = 14;
  }
  optional MessageAT field_15 = 15;

  message MessageAM {
    optional int32 field_1 = 1;
    optional int32 field_2 = 2;
    optional int32 field_3 = 3;
    optional int32 field_4 = 4;
  }

  message MessageAO {
    int32 field_1 = 1;
  }

  message MessageAP {
    int32 field_1 = 1;
  }

  message MessageAQ {
    int32 field_1 = 1;
  }

  message MessageAR {}

  message MessageAS {
    int32 field_1 = 1;
  }

  message MessageAT {
    optional string field_1 = 1;
    optional string field_2 = 2;
  }

  message MessageAN {
    optional string field_1 = 1;
    optional int32 field_2 = 2;
    optional int32 field_3 = 3;
    optional int32 field_4 = 4;
    optional int32 field_5 = 5;
    optional int32 field_6 = 6;
    repeated MessageAU field_7 = 7;
  }

  message MessageAU {
    string field_1 = 1;
    string field_2 = 2;
    int32 field_3 = 3;
    int32 field_4 = 4;
    int32 field_11 = 11;
    optional int32 field_5 = 5;
    optional int32 field_6 = 6;
    optional string field_7 = 7;
    optional string field_8 = 8;
    optional string field_9 = 9;
    optional string field_10 = 10;
  }
}

message MessageL {
  string field_1 = 1;
  string field_2 = 2;
}

message MessageM {
  string field_1 = 1;
  optional string field_2 = 2;
  string field_3 = 3;
}

message MessageN {
  string field_1 = 1;
  string field_2 = 2;
}

message MessageO {
  string field_1 = 1;
  string field_2 = 2;
}

message MessageP {
  string field_1 = 1;
  optional string field_2 = 2;
  optional string field_3 = 3;
  optional string field_4 = 4;
}

[piotrkowalczuk]

One final note: protoc-gen-validate is in maintenance mode.

@rodaine we do not need tons of flexibility, as long as the project will get fixes and compatibility updates we should be fine. Any plans to deprecate protoc-gen-validate?

[rodaine]

protoc-gen-validate is essentially deprecated. It's not going anywhere, and we'll accept security patches, but all active development is focused on the protovalidate ecosystem.

[yordis]

We had to remove the protovalidate since the CEL was causing a OOMKill overtime, we do not large messages as far as I can tell, neither that many descriptors.

@piotrkowalczuk did you find any solution here?

[piotrkowalczuk]

@yordis in our case resource efficiency and predictability are more valuable than cross-language support or quality of life improvements. Therefore we stick to the code generator.

[yordis]

@piotrkowalczuk I just talked to my team lead, and I think we are gonna try to do the same and stick to codegen; we ran out of 4GB just because of this ...

I haven't get into the details, but is there any configuration that would improve this? I am not sure if you dig into the code to see what could be improved or not.

[yordis]

I mantually added WithMessages and I slow down the memory over time, still going up, but much much slower than before. I am gonna way until tomorrow to see how it goes just in case it is "expected"

[piotrkowalczuk]

@yordis, there are a couple of things that can be configured, such as pre-registering descriptors. But nothing related to CEL. That is the largest allocator here.

[rodaine]

To be clear, the memory usage you're seeing is not desirable. A lot of it is originating from the cel-go library, which we're working to mitigate. Changes to support features we wanted to get in prior to v1 prioritized correctness over performance, which is why the memory profile is worse than older versions of the library as well. We're working through perf improvements, both within pv-go and upstream into cel-go, to bring this down to acceptable levels.

[rodaine]

I slow down the memory over time, still going up, but much much slower than before.

@yordis are you saying there's a memory leak? Would love to see some evidence of this if there is so we can address it.

[yordis]

@rodaine I am not sure if it is a memory leak or expected, but this is what we observed in production.

This stopped after I added the manual registration using WithMessages it seems, but obviously, a lot of grunt work now that is easy to miss.

Last 2 weeks

Last 2 days

It seems that the manual registration of the messages helped here, I do not see the grow anymore. Or it is growing slower ... 😭 At least it is not as fast as before (I am talking about the last day)

There is some slow down between 15 to 18 becaused we removed and then we put it back in to test again

Heap Live Object Profiler

This is what I see,

[yordis]

Could be causing CPU spikes as well?

[yordis]

I added WithDisableLazy to see if that helps with the situation, at this point, we may remove the pkg all together and invest some time into proper value objects 😭

[rodaine]

Tackled some low hanging fruit, but there's more still to do.