Eradicating complexity to enhance enterprise efficiency: How Bridgewater Associates constructed a scalable, safe, Spark-based analysis service on AWS


This can be a visitor submit co-written by Sergei Dubinin, Oleksandr Ierenkov, Illia Popov and Joel Thompson, from Bridgewater.

Bridgewater’s core mission is to know how the world works by analyzing the drivers of markets and turning that understanding into high-quality portfolios and funding recommendation for our shoppers. Inside Bridgewater Expertise, we attempt to make our researchers as productive as potential at what they do finest: constructing the basic understanding of world markets. This implies eliminating the necessity to cope with underlying IT infrastructure, and specializing in constructing and bettering their funding concepts.

On this submit, we study our proprietary service in 4 dimensions. We speak about our enterprise challenges, how we met our excessive safety bar, how we will scale to fulfill the calls for of the enterprise, and the way we do all of this in an economical method.


Our researchers’ demand for compute required to develop and check their funding logic is continually rising. This constant and aggressive development in compute capability was a driving drive behind our preliminary determination to maneuver to the general public cloud.

Using the dimensions of the AWS Cloud has allowed us to generate funding indicators and views of the world that will have been unattainable to do on premises. After we first moved this analytical workload to AWS, we constructed on Amazon Elastic Compute Cloud (Amazon EC2) together with different providers equivalent to Elastic Load Balancing, AWS Auto Scaling, and Amazon Easy Storage Service (Amazon S3) to supply core performance. A short while later, we moved to the AWS Nitro System, finishing jobs 20% sooner—permitting our analysis groups to iterate extra shortly on their funding concepts.

The following step in our evolution began 2 years in the past once we adopted Apache Spark because the underlying compute engine for our funding logic execution service. This helped streamline our analytics pipeline, eradicating duplication and decoupling lots of the plugins we have been creating for our researchers. Relatively than run Apache Spark ourselves, we selected Amazon EMR as a hosted Spark platform. Nevertheless, we quickly found that Amazon EMR on EC2 wasn’t a very good match for the best way we wished to make use of it. For instance, we will’t predict when a researcher will submit a job, so to keep away from having our researchers anticipate a model new EMR cluster to be created and bootstrapped, we used long-lived EMR clusters, which pressured many various jobs to run on the identical cluster. Nevertheless, as a result of a single EMR cluster can solely exist in a single Availability Zone, our cluster was restricted to solely with the ability to launch situations in that Availability Zone. On the important scale that we have been working at, particular person Availability Zones began working out of our desired occasion capability to fulfill our wants. Though we may launch many various clusters throughout completely different Availability Zones, that would depart us dealing with job scheduling at a excessive stage, which was the entire level of utilizing Amazon EMR and Spark. Moreover, to be as cost-efficient as potential, we wished to constantly scale the variety of nodes within the cluster primarily based on demand, and consequently, we might churn by 1000’s of nodes a day. This fixed churning of nodes brought about job failures and extra operational overhead for our groups.

We introduced these considerations to AWS, who took the lead in pushing these points to decision. AWS partnered carefully with us to know our use circumstances and the impression of job failures, and tirelessly labored with us to unravel these challenges. Working with the Amazon EMR staff, we narrowed down the issue to our aggressive scaling patterns, which the service couldn’t deal with at the moment. Over the course of just some months, the Amazon EMR staff made a number of service enhancements within the scaling mechanism to fulfill our wants and the wants of many different AWS clients.

Whereas working carefully with the Amazon EMR staff on these points, the AWS staff knowledgeable us of the event of Amazon EMR on EKS, a managed service that will allow us to run Spark workloads on Amazon Elastic Kubernetes Service (Amazon EKS). Amazon EKS is a strategic platform for us throughout numerous enterprise items at Bridgewater, and after doing a proof of idea of our workload utilizing EMR on EKS, it turned clear that this was a greater match for our use case and extra aligned with our strategic course. After migrating to EMR on EKS, we will now make the most of capability in a number of Availability Zones and enhance our resiliency to EMR cluster points or broader service occasions, whereas nonetheless assembly our excessive safety bar.


One other vital facet of our service is making certain it maintains the suitable safety posture. Amongst different considerations, Bridgewater strictly compartmentalizes entry to completely different funding concepts, and we should defend in opposition to the potential of a malicious insider making an attempt to steal our mental property or in any other case hurt Bridgewater. To steadiness the trade-offs between velocity and safety, we designed safety controls to defend in opposition to probably malicious jobs, whereas enabling our researchers to shortly iterate on their code. That is made extra difficult by the design of Spark’s Kubernetes backend. The Spark driver, which in our case is working arbitrary and untrusted code, needs to be given Kubernetes role-based entry management (RBAC) permissions to create Kubernetes Pods. The flexibility to create Pods could be very highly effective and might result in privilege escalation.

Our first layer of isolation is to run every job in its personal Kubernetes namespace (and, subsequently, in its personal EMR on EKS digital cluster). A namespace and digital cluster are created when the job is able to be submitted, and so they’re deleted when that job is completed. This prevents one job from interfering straight with one other job, however there are nonetheless different vectors to defend in opposition to. For instance, Spark drivers shouldn’t be creating Pods with containers that run as root or supply their pictures from unapproved repositories. We first investigated PodSecurityPolicies for this goal. Nevertheless, they couldn’t resolve all of our use circumstances (equivalent to limiting the place container pictures could be pulled from), and they’re at the moment being deprecated and can finally be eliminated. As an alternative, we turned to Open Coverage Agent (OPA) Gatekeeper, which supplies a versatile strategy for writing insurance policies in code that may do extra advanced authorization selections and permits us to implement our desired suite of controls. We additionally labored with the AWS Service Crew so as to add additional protection in depth, equivalent to making certain that each one Pods created by EMR on EKS dropped all Linux capabilities, which we may then implement with Gatekeeper.

The next diagram illustrates how we will preserve the required job separation inside our analysis service.


One of many largest motivations of our evolution to Spark on Amazon EMR after which on EMR on EKS was bettering the effectivity of our useful resource utilization by aggressively scaling primarily based on demand. Our basic cause-and-effect understanding of markets and economies is powered by our systematic, high-performance compute Spark grid. We run simulations at a always rising scale and wish an structure that may scale up and meet our foreseeable enterprise wants for the following a number of years.

Our platform runs two sorts of jobs: advert hoc interactive and scheduled batch. Every kind of job brings its personal scaling complexities, and each benefited from the evolution to EMR on EKS. Advert hoc jobs could be submitted at any time all through enterprise hours, and the simulation determines how a lot compute capability is required. For instance, a specific job may have one EC2 occasion or 100 EC2 situations. This will translate to a whole lot of EC2 situations needing to be spun up or down inside a couple of minutes. The scheduled batch jobs run periodically all through the day with predetermined simulations and equally interprets to a whole lot of EC2 situations spinning up or down. In complete, scaling up and down by many a whole lot of EC2 situations in a couple of minutes is frequent, and we wanted an answer that would meet these enterprise necessities.

For this particular downside, we wanted an answer that was in a position to deal with aggressive scaling occasions on the order of a whole lot of EC2 situations per minute. Moreover, when working at this scale, it’s vital to each diversify occasion varieties and unfold jobs throughout Availability Zones. EMR on EKS empowers us to run fully-managed Spark jobs on an EKS cluster that spans a number of Availability Zones and supplies the choice to decide on a heterogeneous set of occasion varieties for Amazon EKS. Spanning a single EKS cluster throughout Availability Zones allows us to make the most of compute capability throughout all the Area, thereby rising occasion variety and availability for this workload. As a result of Spark jobs are working inside containers on Amazon EKS, we will simply swap out occasion varieties inside the EKS cluster or run completely different occasion varieties inside the similar cluster. On account of these capabilities, we’re in a position to usually scale our manufacturing service to roughly 1,600 EC2 situations totaling 25,000 cores at peak, working 3,000 jobs per day.

Lastly, in late 2021, we performed some scaling exams to see what the real looking limits of our service are. We’re joyful to share that we have been in a position to scale our service to 3 instances our regular each day measurement by way of compute and simulations run. This train has validated that we can meet the rise in enterprise demand with out committing extra engineering sources to take action.

Price administration

Along with considerably rising our skill to scale, we additionally have been in a position to design the answer to be extraordinarily value efficient. Previous to EMR on EKS, we had two choices for Spark jobs: both self-managed on Amazon EC2 or utilizing Amazon EMR on EC2. Self-managing on Amazon EC2 meant that we wanted to handle the complexities of scheduling jobs on nodes, handle the Spark clusters themselves, and develop a separate utility to provision and cease EC2 situations as Spark jobs ran to scale the workloads. Amazon EMR on EC2 supplies a managed service to run Spark workloads on Amazon EC2. Nevertheless, for patrons like us who have to function in a number of Availability Zones and have already got a expertise footprint on Kubernetes, EMR on EKS made extra sense.

Shifting to EMR on EKS allows us to scale dynamically as jobs are submitted, producing enormous value financial savings. Simulation capability is right-sized inside the vary of some minutes; one thing that’s not potential with one other resolution. Moreover, our funding in Amazon EC2 Compute Financial savings Plans supplies us with the financial savings and adaptability to fulfill our wants; we simply have to specify what number of compute hours we’re dedicated to in a specific Area and AWS handles the remainder. You possibly can learn extra about the price advantages of EMR on EKS in Amazon EMR on Amazon EKS supplies as much as 61% decrease prices and as much as 68% efficiency enchancment for Spark workloads.

The long run

Though we’re at the moment assembly our key customers’ wants, we’ve got prioritized a number of enhancements to our service for the long run. First, we plan on changing the Kubernetes Cluster Autoscaler with Karpenter. Given our aggressive and frequent compute scaling, we’ve got discovered that some jobs could be unexpectedly stopped utilizing the Cluster Autoscaler. We expertise this about six instances a day. We count on Karpenter will significantly diminish the prevalence of this failure mode. To be taught extra about Karpenter, try Introducing Karpenter – An Open-Supply Excessive-Efficiency Kubernetes Cluster Autoscaler.

Second, we’re transferring a number of complementary providers which can be at the moment working on EC2 to EKS. It will enhance our skill to deploy significant enhancements for our enterprise and enhance resiliency to service occasions.

Lastly, we’re making long term efforts to enhance our resiliency to regional service occasions. We’re exploring broadening our operations to different AWS Areas, which might enable us to extend our service availability in addition to preserve our burst capability.


Working carefully with AWS groups, we have been in a position to develop a scalable, safe, and cost-optimized service on AWS that permits our researchers to generate bigger and extra advanced funding concepts with out worrying about IT infrastructure. Our service runs our Spark-based simulations throughout a number of Availability Zones at near-full utilization with out having to fret about constructing or sustaining a scheduling platform. Lastly, we’re in a position to meet and surpass our safety benchmarks by creating job separation utilizing native AWS constructs at scale. This has given us super confidence that our mission-critical information is protected within the AWS Cloud.

By way of this shut partnership with AWS, Bridgewater is poised to anticipate and meet the rigorous calls for of our researchers for years to come back; one thing that was not potential in our outdated information facilities or with our prior structure. Our President and CTO, Igor Tsyganskiy, not too long ago spoke with AWS at size on this partnership. For the video of this dialogue, try Merging Enterprise and Tech – Bridgewater’s Information to Drive Agility.


  • Igor Tsyganskiy, President and Chief Expertise Officer, Bridgewater
  • Aaron Linsky, Sr. Product Supervisor, Bridgewater
  • Gopinathan Kannan, Sr. Mgr. Engineering, Amazon Net Providers
  • Vaibhav Sabharwal, Sr. Buyer Options Supervisor, Amazon Net Providers
  • Joseph Marques, Senior Principal Engineer, Amazon Net Providers
  • David Brown, VP EC2, Amazon Net Providers

In regards to the authors

Sergei Dubinin is an Engineering Supervisor with Bridgewater. He’s captivated with constructing huge information processing programs which can be appropriate for a safe, secure, and performant use in manufacturing.

Oleksandr Ierenkov is a Answer Architect for EPAM Techniques. He has targeted on serving to Bridgewater migrate in-house distributed programs to microservices on Kubernetes and numerous AWS-managed providers with a give attention to operational effectivity. Oleksandr is principally the identical title as Alexander, solely Ukrainian.

Anthony Pasquariello is a Senior Options Architect at AWS primarily based in New York Metropolis. He focuses on modernization and safety for our superior enterprise clients. Anthony enjoys writing and talking about all issues cloud. He’s pursuing an MBA, and acquired his MS and BS in Electrical & Laptop Engineering.

Illia Popov is a Tech Lead for EPAM Techniques. Illia has been working with Bridgewater since 2018 and was energetic in planning and implementing the migration to EMR on EKS. He’s excited to proceed delivering worth to Bridgewater by adapting managed providers in shut cooperation with AWS.

Peter Sideris is a Sr. Technical Account Supervisor at AWS. He works with a few of our largest and most advanced clients to make sure their success within the AWS Cloud. Peter enjoys his household, marine reef conserving, and volunteers his time to the Boy Scouts of America in a number of capacities.

Joel Thompson is an Architect at Bridgewater Associates, the place he has labored in a wide range of expertise roles over the previous 13 years, together with constructing among the earliest foundations of AWS adoption at Bridgewater. He’s captivated with fixing difficult issues to securely ship worth to the enterprise. Outdoors of labor, Joel is an avid skier, helped co-found the fwd:cloudsec cloud safety convention, and enjoys touring to spend time with family and friends.


Leave a Reply

Your email address will not be published. Required fields are marked *