Releases: karafka/karafka-core
Releases · karafka/karafka-core
v2.6.2
- [Enhancement] Document that a leaf's
defaultvalue is intentionally shared by reference across the class template and every config instance produced byConfigurable::Node#deep_dup. This uniform rule (the leaf is shallow-copied) is what lets a shared service object passed as a default (e.g. a logger) keep its identity across all configs; the flip side is that an in-place mutation of a mutable container default (config.list << :x) is visible on every instance. Callers that need a per-instance mutable default should assign it inside aconfigureblock or dup it themselves rather than relying on a mutabledefault:(e.g.default: []). Adds characterization tests covering the shared-default behavior. - [Fix]
require "pathname"explicitly inlib/karafka-core.rb(with the other top-level requires).Karafka::Core.gem_rootreturns aPathname, but the gem never requiredpathname-- it only worked because Bundler (or another gem) happened to load it first. In an environment where nothing else loads it,gem_rootraisedNameError: uninitialized constant Pathname. - [Enhancement] Document the
virtualrule result contract: a rule must return a freshly builtArrayof[path, message]error pairs on every call.Contract#calltakes ownership of that array and prepends the current scope onto each pair in place (avoiding a per-error allocation), so returning a memoized, shared or frozen array is unsupported -- the in-place scoping would accumulate the scope prefix across validations or raiseFrozenError. Adds characterization tests for the supported and unsupported patterns. - [Fix]
Configurable::Node#registerraises the documented "already registered"ArgumentErrorfor a name already used by an unread lazy-with-constructor setting. The duplicate guard only checked@configs_refs, but a lazy setting with a constructor is absent from it until first read, soregistersilently overwrote it; it now checks the defined children. - [Fix]
Contractable::Contract.nestednow pops its path in anensure. If the block raised while the contract was being defined and the caller rescued it, the path stayed on the nesting stack and was prefixed onto every rule defined afterwards. - [Fix]
Contract#callno longer raisesNoMethodErrorwhen validating a non-Hash root with a 1-key or 2-key rule path; it reports the path as missing, consistent with the 3+-key path (and the non-Hash intermediate handling added in 2.6.1). - [Fix] Honor
excluded_keyscontaining"cgrp"inStatisticsDecoratoronly_keysmode. Thecgrpbranch of the structure-aware fast path lacked the exclusion guard that thebrokersandtopicsbranches have, so excluding the consumer-group subtree still decorated it (inconsistent with the full-decoration path). - [Fix] Guard the patched rdkafka error callback against a null client pointer. librdkafka can invoke the error callback with a NULL
rd_kafka_t(e.g. very early in client construction); callingrd_kafka_nameon it dereferenced the null pointer and could segfault the process. Mirrors the upstreamErrorCallback. - [Fix] Resolve fatal errors in the patched rdkafka error callback.
ERR__FATALis only a generic marker, so the callback now fetches the real underlying error code and description viaRdkafkaError.build_fatal(rd_kafka_fatal_error) instead of reporting the generic fatal code. Mirrors the upstreamErrorCallback. - [Fix] A lazy setting declared without a constructor (
setting(:x, lazy: true)) no longer raises when its accessor is read.lazy: trueonly makes sense together with a constructor to (re)evaluate; without one there is nothing to evaluate, so such a setting now behaves like a regular setting backed by its default. Previously reading it raisedNoMethodError—nil.aritythrough the dynamic accessor for a falsy default, or a missing accessor for a truthy default. - [Fix]
Contract#callno longer raisesNoMethodErrorwhen a virtual rule returnsfalse. A virtual rule now signals "no errors" with any non-Array result (true/false/nil); only anArrayof error pairs is collected. Previously afalsereturn reachedfalse.each(anilreturn was already tolerated). - [Fix] Manage
CallbacksManagercallbacks copy-on-write:add/deleterebuild and atomically swap an immutable values snapshot under a mutex, and#calliterates that snapshot directly. The previous values cache (introduced in 2.5.11) was lazily invalidated from within#call, which could not be done atomically against a concurrentadd/delete, so a callback racing with dispatch could be silently lost — a removed one kept firing forever, or a newly added one never fired. The copy-on-write read takes no lock and allocates nothing per call, so the race is fixed without reintroducing a per-call#valuesallocation. librdkafka statistics/error callbacks fire from a background thread while callbacks are registered/unregistered, so this was reachable in practice. - [Fix] Manage
Notificationssubscriptions copy-on-write (#subscribe,#unsubscribeand#clearreplace the per-event listener array instead of mutating it in place) so a listener that unsubscribes itself (or another) from within its own handler no longer causes the listener following it to be silently skipped, and concurrent subscribe/unsubscribe during dispatch is safe. Dispatch keeps iterating the live array directly, so there is no per-notification allocation on the hot path. - [Fix] Report a freeze duration (
_fd) of0for statistics keys that are newly introduced in an emission (e.g. a broker or partition that appears mid-stream) instead of the elapsed time since the previous emission. A key that did not exist in the prior emission could not have been "frozen" for any duration, so accumulating the inter-emission gap was incorrect and also made the relatedStatisticsDecoratorspec flaky on slow CIs (_fddepended on the wall-clock gap between the two emissions). - [Fix] Make assigning a setting on a frozen
Configurable::Nodeatomic. The ivar-backed writer evaluated@configs_refs[name] = valuebeforeinstance_variable_set, so a frozen node mutated the canonical store and only then raisedFrozenError, leaving the store and the ivar-backed reader permanently out of sync. It now raises before touching any state. - [Fix]
Configurable::Node#to_hnow evaluates a setting's constructor with its default (arity-aware, matching#compile) instead of calling it with no arguments. The documented->(default) { ... }constructor form previously raisedArgumentError: wrong number of argumentsfrom#to_hwhenever the value was not yet in the config store (e.g.#to_hon an unconfigured instance, or an unread lazy setting). - [Fix] Honor
excluded_keysinsideStatisticsDecoratoronly_keysdecoration. A key listed in bothonly_keysandexcluded_keyswas still decorated because the direct-access decoration loop never consultedexcluded_keys; exclusion now wins, matching the full-decoration path. - [Fix] Strip the tests/specs root directory as an anchored prefix (
sub(/\A.../)) instead of a globalgsubinMinitestLocatorandRSpecLocator. When the root directory string recurred later in a test/spec file path, the global replace removed every occurrence and corrupted the derived subject class path; only the leading prefix is now removed.
v2.6.1
- [Enhancement] Speed up
Contract#callby ~1.25x for minimal and ~1.4x for fully populated data: resolve rule paths with a singleHash#fetchper level instead ofkey?+[], inline the per-rule type dispatch into the rules loop, and compare the dig sentinel via#equal?so#==is never dispatched to the validated (user-provided) values. This is the per-message validation path in WaterDrop producers. - [Fix]
Contract#callwith rule paths of 3+ keys no longer raisesNoMethodErrorwhen an intermediate value is not aHashand reports the path as missing instead, consistent with the 2-key path behavior. - [Change] Reject reserved setting names with an
ArgumentErrorinConfigurable::Node#settingand#register: internal state names (node_name,children,nestings,compiled,configs_refs,local_defs) and the node public API names (setting,configure,to_h,deep_dup,register,compile). Previously such names silently shadowed the node own accessors, breakingdeep_duporto_h, and assignments likeconfig.children = valuecorrupted the node internal state. - [Enhancement] Skip the event name mapping hash lookup in
Monitor#instrumentwhen no namespace is used and the event id is already aString, which is the case for all events in the Karafka ecosystem (~1.2x faster dispatch on the common no-subscribers path). Symbol event ids and namespaced monitors keep going through the mapping. - [Enhancement] Mirror config values into instance variables and use
attr_readerbased readers inConfigurable::Node, yielding ~1.4x faster flat and ~1.6x faster nested settings reads on hot paths.@configs_refsremains the canonical store; non-identifier setting names (e.g. registered names with dashes) keep the previous hash-based accessors. - [Enhancement] Instantiate each
Configurable::Nodethrough a per-layout anonymous subclass so the ivar-backed settings do not grow object shape variations on the sharedNodeclass (which would degrade ivar access and trigger Ruby performance warnings).deep_dupreuses the template's subclass, so duplicated configs share object shapes. - [Fix] Symbolize setting names at definition time (
setting, same asregister) and on config store writes soStringsetting names work end to end (accessors,#to_h, recompilation state) and cannot corrupt node internal state when matching reserved internal names (previously string-named settings were quietly broken as accessors and the store disagreed on the key type). - [Change] Config nodes are now instances of anonymous
Nodesubclasses:is_a?(Karafka::Core::Configurable::Node)still holds, butinstance_of?(Node)is nowfalseandnode.class.nameisnil. - [Change] Assigning a setting on a frozen config node now raises
FrozenError(previously the write silently mutated internal storage despite the freeze).
v2.6.0
- [Enhancement] Add
Node#registerto allow runtime key-value registration on compiled nodes without going through the staticsettingDSL. Useful for dynamic registries (e.g. named clusters) where setting names are not known at class-load time. - [Enhancement] Replace version-gated
Warning[:performance]with aWarning.categories-based loop that enables all opt-in Ruby warning categories automatically, picking up new categories (e.g.strict_unused_blockin Ruby 3.4+) without future patches.
v2.5.13
v2.5.12
v2.5.11
- [Enhancement] Specialize
Contract#digfor common 1-key and 2-key paths to avoid iterator overhead, yielding ~1.5x faster single-key lookups and ~1.45x faster two-key nested lookups. - [Enhancement] Replace
Node#build_accessors@local_defsArray with Hash for O(1) membership checks instead of O(n)Array#include?, yielding up to ~5x faster accessor lookups at 50 settings. - [Enhancement] Use frozen
EMPTY_ARRAYconstant forContract#calland#validate!defaultscopeparameter to avoid allocating a new Array on every invocation, yielding ~1.36x faster call dispatch and saving 1 Array allocation per call. - [Enhancement] Pre-resolve
@events_methods_mapmethod name before the listener notification loop inNotifications#instrumentto avoid repeated Hash lookup per listener, yielding ~1.12x faster event dispatch with multiple listeners. - [Enhancement] Cache a frozen success
Resultsingleton viaResult.successto eliminate 1 object allocation per successfulContract#callon the happy path. - [Enhancement] Skip nestings block re-evaluation in
Node#deep_dupto avoid recreating children that are immediately overwritten, yielding ~14x faster deep_dup and reducing allocations from ~620 to ~66 objects for large configs. - [Enhancement] Cache
CallbacksManager#callvalues snapshot and invalidate onadd/deleteto avoid allocating a new Array on every invocation while preserving thread-safety snapshot semantics, saving 1 Array allocation per call. - [Enhancement] Store execution time separately in
Eventand build the merged payload hash lazily on#payloadaccess, eliminating 1 Hash allocation perNotifications#instrumentcall when listeners use#[]access (the common pattern), yielding ~1.7x faster event dispatch. - [Enhancement] Replace
StatisticsDecorator#diffpending-writes buffer withkeys.eachdirect-write iteration, eliminating the buffer and write-back loop for ~13% faster decoration at scale (10 brokers, 20 topics, 2000 partitions). - [Enhancement] Reorder
StatisticsDecorator#difftype checks to testNumericbeforeHash, matching the ~80% numeric value distribution in librdkafka statistics. - [Enhancement] Support
only_keysoption inStatisticsDecoratorto decorate only specified numeric keys (e.g.consumer_lag,committed_offset). When combined withexcluded_keys, reduces decoration cost from ~80ms to ~8.5ms per call on large clusters (10 brokers, 20 topics, 2000 partitions) by using structure-aware navigation of the librdkafka statistics tree and direct key access instead of full-hash iteration.
v2.5.10
v2.5.9
- [Enhancement] Optimize
StatisticsDecoratorto eliminate per-hash Array allocations by usingeach_pairwith a per-call pending-writes buffer instead ofcurrent.keys.each, reducing allocations from tens of thousands to one per call at scale. - [Enhancement] Inline
StatisticsDecorator#appendand#suffix_keys_forinto#diffto reduce method call overhead by ~96% (from ~915k to ~39k calls at 6400 partitions). - [Enhancement] Support
excluded_keysoption inStatisticsDecoratorto skip decoration of unused librdkafka statistics subtrees (e.g. broker toppars, window stats), yielding ~28% additional speedup at scale.
v2.5.8
v2.5.7
- [Enhancement] Optimize
Node#deep_dupto reduce array allocations during configuration deep copying. - [Enhancement] Optimize
Result#initializeto use sharedEMPTY_HASHconstant for successful validations. - [Enhancement] Optimize
StatisticsDecorator#appendstring concatenation to reduce allocations. - [Enhancement] Optimize
Notifications#instrumentpayload handling to avoid unnecessary hash merges for empty payloads. - [Change] Remove Ruby 3.1 support due to EOL.