Characteristic Deep Dive: Watermarking in Apache Spark Structured Streaming

Key Takeaways

  • Watermarks assist Spark perceive the processing progress based mostly on occasion time, when to provide windowed aggregates and when to trim the aggregations state
  • When becoming a member of streams of information, Spark, by default, makes use of a single, world watermark that evicts state based mostly on the minimal occasion time seen throughout the enter streams
  • RocksDB might be leveraged to cut back strain on cluster reminiscence and GC pauses
  • StreamingQueryProgress and StateOperatorProgress objects comprise key details about how watermarks have an effect on your stream


When constructing real-time pipelines, one of many realities that groups should work with is that distributed knowledge ingestion is inherently unordered. Moreover, within the context of stateful streaming operations, groups want to have the ability to correctly monitor occasion time progress within the stream of information they’re ingesting for the correct calculation of time-window aggregations and different stateful operations. We will clear up for all of this utilizing Structured Streaming.

For instance, let’s say we’re a workforce engaged on constructing a pipeline to assist our firm do proactive upkeep on our mining machines that we lease to our prospects. These machines all the time must be working in high situation so we monitor them in real-time. We might want to carry out stateful aggregations on the streaming knowledge to know and determine issues within the machines.

That is the place we have to leverage Structured Streaming and Watermarking to provide the mandatory stateful aggregations that may assist inform choices round predictive upkeep and extra for these machines.

What Is Watermarking?

Usually talking, when working with real-time streaming knowledge there will likely be delays between occasion time and processing time attributable to how knowledge is ingested and whether or not the general utility experiences points like downtime. Because of these potential variable delays, the engine that you just use to course of this knowledge must have some mechanism to resolve when to shut the combination home windows and produce the combination outcome.

Whereas the pure inclination to treatment these points could be to make use of a set delay based mostly on the wall clock time, we’ll present on this upcoming instance why this isn’t the most effective answer.

To clarify this visually let’s take a situation the place we’re receiving knowledge at numerous occasions from round 10:50 AM → 11:20 AM. We’re creating 10-minute tumbling home windows that calculate the typical of the temperature and strain readings that got here in throughout the windowed interval.

On this first image, now we have the tumbling home windows set off at 11:00 AM, 11:10 AM and 11:20 AM resulting in the outcome tables proven on the respective occasions. When the second batch of information comes round 11:10 AM with knowledge that has an occasion time of 10:53 AM this will get integrated into the temperature and strain averages calculated for the 11:00 AM → 11:10 AM window that closes at 11:10 AM, which doesn’t give the proper outcome.

Visual representation of a Structured Streaming pipeline ingesting batches of temperature and pressure data

To make sure we get the proper outcomes for the aggregates we wish to produce, we have to outline a watermark that may enable Spark to know when to shut the combination window and produce the proper combination outcome.

In Structured Streaming functions, we will make sure that all related knowledge for the aggregations we wish to calculate is collected through the use of a function referred to as watermarking. In probably the most primary sense, by defining a watermark Spark Structured Streaming then is aware of when it has ingested all knowledge as much as a while, T, (based mostly a set lateness expectation) in order that it will possibly shut and produce windowed aggregates as much as timestamp T.

This second visible exhibits the impact of implementing a watermark of 10 minutes and utilizing Append mode in Spark Structured Streaming.

Visual representation of the effect a 10-minute watermark has when applied to the Structured Streaming pipeline.

Not like the primary situation the place Spark will emit the windowed aggregation for the earlier ten minutes each ten minutes (i.e. emit the 11:00 AM →11:10 AM window at 11:10 AM), Spark now waits to shut and output the windowed aggregation as soon as the max occasion time seen minus the desired watermark is larger than the higher certain of the window.

In different phrases, Spark wanted to attend till it noticed knowledge factors the place the newest occasion time seen minus 10 minutes was higher than 11:00 AM to emit the ten:50 AM → 11:00 AM combination window. At 11:00 AM, it doesn’t see this, so it solely initializes the combination calculation in Spark’s inner state retailer. At 11:10 AM, this situation remains to be not met, however now we have a brand new knowledge level for 10:53 AM so the inner state will get up to date, simply not emitted. Then lastly by 11:20 AM Spark has seen a knowledge level with an occasion time of 11:15 AM and since 11:15 AM minus 10 minutes is 11:05 AM which is later than 11:00 AM the ten:50 AM → 11:00 AM window might be emitted to the outcome desk.

This produces the proper outcome by correctly incorporating the info based mostly on the anticipated lateness outlined by the watermark. As soon as the outcomes are emitted the corresponding state is faraway from the state retailer.

Incorporating Watermarking into Your Pipelines

To know easy methods to incorporate these watermarks into our Structured Streaming pipelines, we’ll discover this situation by strolling by way of an precise code instance based mostly on our use case acknowledged within the introduction part of this weblog.

Let’s say we’re ingesting all our sensor knowledge from a Kafka cluster within the cloud and we wish to calculate temperature and strain averages each ten minutes with an anticipated time skew of ten minutes. The Structured Streaming pipeline with watermarking would appear to be this:


sensorStreamDF = spark 
  .possibility("kafka.bootstrap.servers", "host1:port1,host2:port2") 
  .possibility("subscribe", "tempAndPressureReadings") 

sensorStreamDF = sensorStreamDF 
.withWatermark("eventTimestamp", "10 minutes") 
.groupBy(window(sensorStreamDF.eventTimestamp, "10 minutes")) 

  .possibility("checkpointLocation", "/delta/occasions/_checkpoints/temp_pressure_job/")

Right here we merely learn from Kafka, apply our transformations and aggregations, then write out to Delta Lake tables which will likely be visualized and monitored in Databricks SQL. The output written to the desk for a specific pattern of information would appear to be this:

Output from the streaming query defined in PySpark code sample above

To include watermarking we first wanted to determine two gadgets:

  1. The column that represents the occasion time of the sensor studying
  2. The estimated anticipated time skew of the info

Taken from the earlier instance, we will see the watermark outlined by the .withWatermark() technique with the eventTimestamp column used because the occasion time column and 10 minutes to symbolize the time skew that we anticipate.


sensorStreamDF = sensorStreamDF 
.withWatermark("eventTimestamp", "10 minutes") 
.groupBy(window(sensorStreamDF.eventTimestamp, "10 minutes")) 

Now that we all know easy methods to implement watermarks in our Structured Streaming pipeline, it is going to be vital to know how different gadgets like streaming be part of operations and managing state are affected by watermarks. Moreover, as we scale our pipelines there will likely be key metrics our knowledge engineers will want to concentrate on and monitor to keep away from efficiency points. We’ll discover all of this as we dive deeper into watermarking.

Watermarks in Completely different Output Modes

Earlier than we dive deeper, you will need to perceive how your selection of output mode impacts the conduct of the watermarks you set.

Watermarks can solely be used if you find yourself working your streaming utility in append or replace output modes. There’s a third output mode, full mode, by which the whole outcome desk is written to storage. This mode can’t be used as a result of it requires all combination knowledge to be preserved, and therefore can not use watermarking to drop intermediate state.

The implication of those output modes within the context of window aggregation and watermarks is that in ‘append’ mode an combination might be produced solely as soon as and cannot be up to date. Subsequently, as soon as the combination is produced, the engine can delete the combination’s state and thus hold the general aggregation state bounded. Late information – those for which the approximate watermark heuristic didn’t apply (they had been older than the watermark delay interval), due to this fact should be dropped by necessity – the combination has been produced and the combination state deleted.

Inversely, for ‘replace’ mode, the combination might be produced repeatedly ranging from the primary document and on every obtained document, thus a watermark is elective. The watermark is just helpful for trimming the state as soon as heuristically the engine is aware of that no extra information for that combination might be obtained. As soon as the state is deleted, once more any late information should be dropped as the combination worth has been misplaced and may’t be up to date.

You will need to perceive how state, late-arriving information, and the totally different output modes might result in totally different behaviors of your utility working on Spark. The primary takeaway right here is that in each append and replace modes, as soon as the watermark signifies that every one knowledge is obtained for an combination time window, the engine can trim the window state. In append mode the combination is produced solely on the closing of the time window plus the watermark delay whereas in replace mode it’s produced on each replace to the window.

Lastly, by growing your watermark delay window you’ll trigger the pipeline to attend longer for knowledge and probably drop much less knowledge – increased precision, but additionally increased latency to provide the aggregates. On the flip aspect, smaller watermark delay results in decrease precision but additionally decrease latency to provide the aggregates.

Window Delay Size Precision Latency
Longer Delay Window Larger Precision Larger Latency
Shorter Delay Window Decrease Precision Decrease Latency

Deeper Dive into Watermarking

Joins and Watermarking

There are a pair concerns to concentrate on when doing be part of operations in your streaming functions, particularly when becoming a member of two streams. Let’s say for our use case, we wish to be part of the streaming dataset about temperature and strain readings with extra values captured by different sensors throughout the machines.

There are three overarching kinds of stream-stream joins that may be carried out in Structured Streaming: internal, outer, and semi joins. The primary drawback with doing joins in streaming functions is that you will have an incomplete image of 1 aspect of the be part of. Giving Spark an understanding of when there are not any future matches to anticipate is much like the sooner drawback with aggregations the place Spark wanted to know when there have been no new rows to include into the calculation for the aggregation earlier than emitting it.

To permit Spark to deal with this, we will leverage a mixture of watermarks and event-time constraints inside the be part of situation of the stream-stream be part of. This mix permits Spark to filter out late information and trim the state for the be part of operation by way of a time vary situation on the be part of. We show this within the instance beneath:


sensorStreamDF = spark.readStream.format("delta").desk("sensorData")
tempAndPressStreamDF = spark.readStream.format("delta").desk("tempPressData")

sensorStreamDF_wtmrk = sensorStreamDF.withWatermark("timestamp", "5 minutes")
tempAndPressStreamDF_wtmrk = tempAndPressStreamDF.withWatermark("timestamp", "5 minutes")

joinedDF = tempAndPressStreamDF_wtmrk.alias("t").be part of(
   s.sensor_id == t.sensor_id AND
   s.timestamp >= t.timestamp AND

Nonetheless, in contrast to the above instance, there will likely be occasions the place every stream might require totally different time skews for his or her watermarks. On this situation, Spark has a coverage for dealing with a number of watermark definitions. Spark maintains one world watermark that's based mostly on the slowest stream to make sure the best quantity of security in the case of not lacking knowledge.

Builders do have the flexibility to vary this conduct by altering spark.sql.streaming.multipleWatermarkPolicy to max; nonetheless, which means that knowledge from the slower stream will likely be dropped.

To see the total vary of be part of operations that require or might leverage watermarks take a look at this part of Spark's documentation.

Monitoring and Managing Streams with Watermarks

When managing a streaming question the place Spark might have to handle thousands and thousands of keys and hold state for every of them, the default state retailer that comes with Databricks clusters will not be efficient. You may begin to see increased reminiscence utilization, after which longer rubbish assortment pauses. These will each impede the efficiency and scalability of your Structured Streaming utility.

That is the place RocksDB is available in. You'll be able to leverage RocksDB natively in Databricks by enabling it like so within the Spark configuration:


This may enable the cluster working the Structured Streaming utility to leverage RocksDB which may extra effectively handle state within the native reminiscence and benefit from the native disk/SSD as an alternative of maintaining all state in reminiscence.

Past monitoring reminiscence utilization and rubbish assortment metrics, there are different key indicators and metrics that must be collected and tracked when coping with Watermarking and Structured Streaming. To entry these metrics you may have a look at the StreamingQueryProgress and the StateOperatorProgress objects. Take a look at our documentation for examples of easy methods to use these right here.

Within the StreamingQueryProgress object, there's a technique referred to as "eventTime" that may be referred to as and that may return the max, min, avg, and watermark timestamps. The primary three are the max, min, and common occasion time seen in that set off. The final one is the watermark used within the set off.

Abbreviated Instance of a StreamingQueryProgress object

  "id" : "f4311acb-15da-4dc3-80b2-acae4a0b6c11",
  . . . .
  "eventTime" : {
    "avg" : "2021-02-14T10:56:06.000Z",
    "max" : "2021-02-14T11:01:06.000Z",
    "min" : "2021-02-14T10:51:06.000Z",
    "watermark" : "2021-02-14T10:41:06.000Z"
  "stateOperators" : [ {
    "operatorName" : "stateStoreSave",
    "numRowsTotal" : 7,
    "numRowsUpdated" : 0,
    "allUpdatesTimeMs" : 205,
    "numRowsRemoved" : 0,
    "allRemovalsTimeMs" : 233,
    "commitTimeMs" : 15182,
    "memoryUsedBytes" : 91504,
    "numRowsDroppedByWatermark" : 0,
    "numShufflePartitions" : 200,
    "numStateStoreInstances" : 200,
    "customMetrics" : {
      "loadedMapCacheHitCount" : 4800,
      "loadedMapCacheMissCount" : 0,
      "stateOnCurrentVersionSizeBytes" : 25680
  . . . .

These pieces of information can be used to reconcile the data in the result tables that your streaming queries are outputting and also be used to verify that the watermark being used is the intended eventTime timestamp. This can become important when you are joining streams of data together.

Within the StateOperatorProgress object there is the numRowsDroppedByWatermark metric. This metric will show how many rows are being considered too late to be included in the stateful aggregation. Note that this metric is measuring rows dropped post-aggregation and not the raw input rows, so the number is not precise but can give an indication that there is late data being dropped. This, in conjunction with the information from the StreamingQueryProgress object, can help developers determine whether the watermarks are correctly configured.

Multiple Aggregations, Streaming, and Watermarks

One remaining limitation of Structured Streaming queries is chaining multiple stateful operators (e.g. aggregations, streaming joins) in a single streaming query. This limitation of a singular global watermark for stateful aggregations is something that we at Databricks are working on a solution for and will be releasing more information about in the coming months. Check out our blog on Project Lightspeed to learn more: Project Lightspeed: Faster and Simpler Stream Processing With Apache Spark (


With Structured Streaming and Watermarking on Databricks, organizations, like the one with the use case described above, can build resilient real-time applications that ensure metrics driven by real-time aggregations are being accurately calculated even if data is not properly ordered or on-time. To learn more about how you can build real-time applications with Databricks, contact your Databricks representative.

Similar Posts

Leave a Reply

Your email address will not be published.