Running stream mode as a cluster

[ploef]

Hi,

Are there any plans for making it possible to scale Bento horizontally in stream mode? Running Bento using stream mode is really great but having one instance for running multiple pipelines is a single point of failure and it it would be great if there would be an option to create a cluster.

[jem-davies]

Hey @ploef - this does come up from time to time ... You are correct about the limitations regarding streams mode & horizontally scaling.

I have thought in the past that one way to do it would be to create a controller, that kept a record of what pods where running what streams, and have that server as a proxy to the actual Bento containers.

Be advised that there is no current plan to do such a thing.

Depending on the use-case (or rather what upstream system) you have - there are ways to achieve such a thing - but the ability to do so is dictated by upstream systems not Bento.

[ploef]

Hi @jem-davies - thanks for the response! The upstream system is a timescaledb database at the moment (could be something else later like webhook or S3 e.g). The streams mode suites our use-case very well because it offers business users to manage their own pipelines relatively easy. I'm worried a bit about the performance though.