We recently shipped an update that fully integrates RunReveal’s destinations product with our Pipelines product, allowing our customers to selectively route data to specific destinations. This is an update to our destinations feature that was previously all-or-nothing with it’s routing capabilities.

We’re excited to show you how this works and share more about the implementation.

Creating a destination

Prior to this change, an object storage destination would route all data to that destination. To show this in action, I'll create a Cloudflare R2 bucket and walk through routing a specific subset of data to it.

Start by creating the bucket in Cloudflare. It's a great choice for this kind of selective routing since egress fees are free.

Once your bucket is ready, creating an R2 destination in RunReveal is straightforward. Each destination type has slightly different configuration requirements; for R2, you just need your endpoint URL, bucket name, and R2 Access + Secret key.

Within Destinations in RunReveal, you can also configure batch sizes, flush settings, and other data-centric options for this routing step. We plan to add additional data formats beyond line-delimited JSON, including Parquet, soon.

Creating a pipeline and selectively routing to a destination

In Pipelines, create a new topic that matches on the source type of your identity provider logs, or whatever subset of logs you intend to route to your object storage destination.

As a refresher, topics are used to match and figure out what pipeline should be invoked.

Your pipeline will be pre-populated with default steps. From there, just add a "Send to Destination" step and select your object storage destination.

You can also add further matching steps within the pipeline to narrow the subset of logs being routed. You should notice that the logs will begin being routed to your bucket immediately!

What’s next

I hinted that something may be coming, and we couldn’t be more excited about how interconnected our platform is becoming. RunReveal wants to make this data accessible from the rest of our features too.

We also know the secret to operationalizing AI is providing it with good data. These types of data capabilities are so important to be done correctly, and within a single platform to limit compatibility and data formatting issues. More soon.