Introducing AWS Glue interactive periods for Jupyter

Interactive Periods for Jupyter is a brand new pocket book interface within the AWS Glue serverless Spark atmosphere. Beginning in seconds and routinely stopping compute when idle, interactive periods present an on-demand, highly-scalable, serverless Spark backend to Jupyter notebooks and Jupyter-based IDEs corresponding to Jupyter Lab, Microsoft Visible Studio Code, JetBrains PyCharm, and extra. Interactive periods exchange AWS Glue improvement endpoints for interactive job improvement with AWS Glue and presents the next advantages:

  • No clusters to provision or handle
  • No idle clusters to pay for
  • No up-front configuration required
  • No useful resource competition for a similar improvement atmosphere
  • Simple set up and utilization
  • The very same serverless Spark runtime and platform as AWS Glue extract, remodel, and cargo (ETL) jobs

Getting began with interactive periods for Jupyter

Putting in interactive periods is straightforward and solely takes a couple of terminal instructions. After you put in it, you may run interactive periods anytime inside seconds of deciding to run. Within the following sections, we stroll you thru set up on macOS and getting began in Jupyter.

To get began with interactive periods for Jupyter on Home windows, observe the directions in Getting began with AWS Glue interactive periods.


These directions assume you’re operating Python 3.6 or later and have the AWS Command Line Interface (AWS CLI) correctly operating and configured. You employ the AWS CLI to make API calls to AWS Glue. For extra data on putting in the AWS CLI, consult with Putting in or updating the newest model of the AWS CLI.

Set up AWS Glue interactive periods on macOS and Linux

To put in AWS Glue interactive periods, full the next steps:

  1. Open a terminal and run the next to put in and improve Jupyter, Boto3, and AWS Glue interactive periods from PyPi. If desired, you may set up Jupyter Lab as an alternative of Jupyter.
    pip3 set up --user --upgrade jupyter boto3 aws-glue-sessions

  2. Run the next instructions to establish the bundle set up location and set up the AWS Glue PySpark and AWS Glue Spark Jupyter kernels with Jupyter:
    SITE_PACKAGES=$(pip3 present aws-glue-sessions | grep Location | awk '{print $2}')
    jupyter kernelspec set up $SITE_PACKAGES/aws_glue_interactive_sessions_kernel/glue_pyspark
    jupyter kernelspec set up $SITE_PACKAGES/aws_glue_interactive_sessions_kernel/glue_spark

  3. To validate your set up, run the next command:

Within the output, you must see each the AWS Glue PySpark and the AWS Glue Spark kernels listed alongside the default Python3 kernel. It ought to look one thing like the next:

Out there kernels:
  Python3		~/.venv/share/jupyter/kernels/python3
  glue_pyspark    /usr/native/share/jupyter/kernels/glue_pyspark
  glue_spark      /usr/native/share/jupyter/kernels/glue_spark

Select and put together IAM principals

Interactive periods use two AWS Identification and Entry Administration (IAM) principals (consumer or function) to perform. The primary is used to name the interactive periods APIs and is probably going the identical consumer or function that you just use with the AWS CLI. The second is GlueServiceRole, the function that AWS Glue assumes to run your session. This is identical function as AWS Glue jobs; in case you’re creating a job along with your pocket book, you must use the identical function for each interactive periods and the job you create.

Put together the shopper consumer or function

Within the case of native improvement, the primary function is already configured in case you can run the AWS CLI. If you happen to can’t run the AWS CLI, observe these steps for establishing. If you happen to usually use the AWS CLI or Boto3 to work together with AWS Glue and have full AWS Glue permissions, you may doubtless skip this step.

  1. To validate this primary consumer or function is ready up, open a brand new terminal window and run the next code:
    aws sts get-caller-identity

    It’s best to see a response like the next. If not, you might not have permissions to name AWS Safety Token Service (AWS STS), otherwise you don’t have the AWS CLI arrange correctly. If you happen to merely get entry denied calling AWS STS, you might proceed if you already know your consumer or function and its wanted permissions.

        "Account": "123456789123",
        "Arn": "arn:aws:iam::123456789123:consumer/MyIAMUser"
        "Account": "123456789123",
        "Arn": "arn:aws:iam::123456789123:function/myIAMRole"

  2. Guarantee your IAM consumer or function can name the AWS Glue interactive periods APIs by attaching the AWSGlueConsoleFullAccess managed IAM coverage to your function.

In case your caller id returned a consumer, run the next:

aws iam attach-user-policy --role-name <myIAMUser> --policy-arn arn:aws:iam::aws:coverage/AWSGlueConsoleFullAccess

In case your caller id returned a job, run the next:

aws iam attach-role-policy --role-name, --policy-arn arn:aws:iam::aws:coverage/AWSGlueConsoleFullAccess

Put together the AWS Glue service function for interactive periods

You may specify the second principal, GlueServiceRole, both within the pocket book itself through the use of the %iam_role magic or saved alongside the AWS CLI config. When you have a job that you just usually use with AWS Glue jobs, this can be that function. If you happen to don’t have a job you employ for AWS Glue jobs, consult with Organising IAM permissions for AWS Glue to set one up.

To set this function because the default function for interactive periods, edit the AWS CLI credentials file and add glue_role_arn to the profile you propose to make use of.

  1. With a textual content editor, open ~/.aws/credentials.
    On Home windows, use C:Usersusername.awscredentials.
  2. Search for the profile you employ for AWS Glue; in case you don’t use a profile, you’re in search of [Default].
  3. Add a line within the profile for the function you propose to make use of like, glue_role_arn=<AWSGlueServiceRole>.
  4. I like to recommend including a default Area to your profile if one shouldn’t be specified already. You are able to do so by including the road area=us-east-1, changing us-east-1 along with your desired Area.
    If you happen to don’t add a Area to your profile, you’re required to specify the Area on the prime of every pocket book with the %area magic.When completed, your config ought to look one thing like the next:

  5. Save the config.

Begin Jupyter and an AWS Glue PySpark pocket book

To begin Jupyter and your pocket book, full the next steps:

  1. Run the next command in your terminal to open the Jupyter pocket book in your browser:

    Your browser ought to open and also you’re offered with a web page that appears like the next screenshot.

  2. On the New menu, select Glue PySpark.

A brand new tab opens with a clean Jupyter pocket book utilizing the AWS Glue PySpark kernel.

Configure your pocket book with magics

AWS Glue interactive periods are configured with Jupyter magics. Magics are small instructions prefixed with % initially of Jupyter cells that present shortcuts to regulate the atmosphere. In AWS Glue interactive periods, magics are used for all configuration wants, together with:

  • %area – Area
  • %profile – AWS CLI profile
  • %iam_role – IAM function for the AWS Glue service function
  • %worker_type – Employee kind
  • %number_of_workers – Variety of employees
  • %idle_timeout – How lengthy to permit a session to idle earlier than stopping it
  • %additional_python_modules – Python libraries to put in from pip

Magics are positioned initially of your first cell, earlier than your code, to configure AWS Glue. To find all of the magics of interactive periods, run %assist in a cell and a full listing is printed. Apart from %%sql, operating a cell of solely magics doesn’t begin a session, however units the configuration for the session that begins subsequent if you run your first cell of code. For this put up, we use three magics to configure AWS Glue with model 2.0 and two G.2X employees. Let’s enter the next magics into our first cell and run it:

%glue_version 2.0
%number_of_workers 2
%worker_type G.2X
%idle_tiemout 60

Welcome to the Glue Interactive Periods Kernel
For extra data on out there magic instructions, please kind %assist in any new cell.

Please view our Getting Began web page to entry essentially the most up-to-date data on the Interactive Periods kernel:
Setting Glue model to: 2.0
Earlier variety of employees: 5
Setting new variety of employees to: 2
Earlier employee kind: G.1X
Setting new employee kind to: G.2X

Whenever you run magics, the output lets us know the values we’re altering together with their earlier settings. Explicitly setting all of your configuration in magics helps guarantee constant runs of your pocket book each time and is really useful for manufacturing workloads.

Run your first code cell and creator your AWS Glue pocket book

Subsequent, we run our first code cell. That is when a session is provisioned to be used with this pocket book. When interactive periods are correctly configured inside an account, the session is totally remoted to this pocket book. If you happen to open one other pocket book in a brand new tab, it will get its personal session by itself remoted compute. Run your code cell as follows:

from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext

sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

Authenticating with profile=default
glue_role_arn outlined by consumer: arn:aws:iam::123456789123:function/AWSGlueServiceRoleForSessions
Making an attempt to make use of current AssumeRole session credentials.
Attempting to create a Glue session for the kernel.
Employee Sort: G.2X
Variety of Employees: 2
Session ID: 12345678-12fa-5315-a234-567890abcdef
Making use of the next default arguments:
--glue_kernel_version 0.31
--enable-glue-datacatalog true
Ready for session 12345678-12fa-5315-a234-567890abcdef to get into prepared standing...
Session 12345678-12fa-5315-a234-567890abcdef has been created

Whenever you ran the primary cell containing code, Jupyter invoked interactive periods, provisioned an AWS Glue cluster, and despatched the code to AWS Glue Spark. The pocket book was given a session ID, as proven within the previous code. We are able to additionally see the properties used to provision AWS Glue, together with the IAM function that AWS Glue used to create the session, the variety of employees and their kind, and every other choices that had been handed as a part of the creation.

Interactive periods routinely initialize a Spark session as spark and SparkContext as sc; having Spark able to go saves lots of boilerplate code. Nonetheless, if you wish to convert your pocket book to a job, spark and sc should be initialized and declared explicitly.

Work within the pocket book

Now that now we have a session up, let’s do some work. On this train, we take a look at inhabitants estimates from the AWS COVID-19 dataset, clear them up, and write the outcomes a desk.

This walkthrough makes use of knowledge from the COVID-19 knowledge lake.

To make the info from the AWS COVID-19 knowledge lake out there within the Knowledge Catalog in your AWS account, create an AWS CloudFormation stack utilizing the next template.

If you happen to’re signed in to your AWS account, deploy the CloudFormation stack by clicking the next Launch stack button:


It fills out many of the stack creation type for you. All you might want to do is select Create stack. For directions on making a CloudFormation stack, see Get began.

After I’m engaged on a brand new knowledge integration course of, the very first thing I usually do is establish and preview the datasets I’m going to work on. If I don’t recall the precise location or desk identify, I usually open the AWS Glue console and search or browse for the desk then return to my pocket book to preview it. With interactive periods, there’s a faster strategy to browse the Knowledge Catalog. We are able to use the %%sql magic to point out databases and tables with out leaving the pocket book. For this instance, the inhabitants desk I would like in is the COVID-19 dataset however I don’t recall its precise identify, so I take advantage of the %%sql magic to look it up:

present tables in `covid-19`  # Keep in mind, dashes in names should be escaped with backticks.

|database|           tableName|isTemporary|
|covid-19|alleninstitute_co...|      false|
|covid-19|alleninstitute_me...|      false|
|covid-19|aspirevc_crowd_tr...|      false|
|covid-19|aspirevc_crowd_tr...|      false|
|covid-19|cdc_moderna_vacci...|      false|
|covid-19|cdc_pfizer_vaccin...|      false|
|covid-19|       country_codes|      false|
|covid-19|  county_populations|      false|
|covid-19|covid_knowledge_g...|      false|
|covid-19|covid_knowledge_g...|      false|
|covid-19|covid_knowledge_g...|      false|
|covid-19|covid_knowledge_g...|      false|
|covid-19|covid_knowledge_g...|      false|
|covid-19|covid_knowledge_g...|      false|
|covid-19|covid_testing_sta...|      false|
|covid-19|covid_testing_us_...|      false|
|covid-19|covid_testing_us_...|      false|
|covid-19|      covidcast_data|      false|
|covid-19|  covidcast_metadata|      false|
|covid-19|enigma_aggregatio...|      false|
solely displaying prime 20 rows

Wanting via the returned listing, we see a desk named county_populations. Let’s choose from this desk, sorting for the most important counties by inhabitants:

choose * from `covid-19`.county_populations type by `inhabitants estimate 2018` desc restrict 10

|            id|  id2|         county|      state|inhabitants estimate 2018|
|            Id|  Id2|         County|      State|    Inhabitants Estima...|
|0500000US01085| 1085|        Lowndes|    Alabama|                    9974|
|0500000US06057| 6057|         Nevada| California|                   99696|
|0500000US29189|29189|      St. Louis|   Missouri|                  996945|
|0500000US22021|22021|Caldwell Parish|  Louisiana|                    9960|
|0500000US06019| 6019|         Fresno| California|                  994400|
|0500000US28143|28143|         Tunica|Mississippi|                    9944|
|0500000US05051| 5051|        Garland|   Arkansas|                   99154|
|0500000US29079|29079|         Grundy|   Missouri|                    9914|
|0500000US27063|27063|        Jackson|  Minnesota|                    9911|

Our question returned knowledge however in an surprising order. It appears to be like like inhabitants estimate 2018 sorted lexicographically if the values had been strings. Let’s use an AWS Glue DynamicFrame to get the schema of the desk and confirm the difficulty:

# Create a DynamicFrame of county_populations and print it is schema
dyf = glueContext.create_dynamic_frame.from_catalog(
    database="covid-19", table_name="county_populations"

|-- id: string
|-- id2: string
|-- county: string
|-- state: string
|-- inhabitants estimate 2018: string

The schema exhibits inhabitants estimate 2018 to be a string, which is why our column isn’t sorting correctly. We are able to use the apply_mapping remodel in our subsequent cell to appropriate the column kind. In the identical remodel, we additionally clear up the column names and different column sorts: clarifying the excellence between id and id2, eradicating areas from inhabitants estimate 2018 (conforming to Hive’s requirements), and casting id2 as an integer for correct sorting. After validating the schema, we present the info with the brand new schema:

# Rename id2 to simple_id and convert to Int
# Take away areas and rename inhabitants est. and convert to Lengthy
mapped = dyf.apply_mapping(
        ("id", "string", "id", "string"),
        ("id2", "string", "simple_id", "int"),
        ("county", "string", "county", "string"),
        ("state", "string", "state", "string"),
        ("population estimate 2018", "string", "population_est_2018", "long"),
|-- id: string
|-- simple_id: int
|-- county: string
|-- state: string
|-- population_est_2018: lengthy

mapped_df = mapped.toDF()

|            id|simple_id|   county|  state|population_est_2018|
|0500000US01001|     1001|  Autauga|Alabama|              55601|
|0500000US01003|     1003|  Baldwin|Alabama|             218022|
|0500000US01005|     1005|  Barbour|Alabama|              24881|
|0500000US01007|     1007|     Bibb|Alabama|              22400|
|0500000US01009|     1009|   Blount|Alabama|              57840|
|0500000US01011|     1011|  Bullock|Alabama|              10138|
|0500000US01013|     1013|   Butler|Alabama|              19680|
|0500000US01015|     1015|  Calhoun|Alabama|             114277|
|0500000US01017|     1017| Chambers|Alabama|              33615|
|0500000US01019|     1019| Cherokee|Alabama|              26032|
|0500000US01021|     1021|  Chilton|Alabama|              44153|
|0500000US01023|     1023|  Choctaw|Alabama|              12841|
|0500000US01025|     1025|   Clarke|Alabama|              23920|
|0500000US01027|     1027|     Clay|Alabama|              13275|
|0500000US01029|     1029| Cleburne|Alabama|              14987|
|0500000US01031|     1031|   Espresso|Alabama|              51909|
|0500000US01033|     1033|  Colbert|Alabama|              54762|
|0500000US01035|     1035|  Conecuh|Alabama|              12277|
|0500000US01037|     1037|    Coosa|Alabama|              10715|
|0500000US01039|     1039|Covington|Alabama|              36986|
solely displaying prime 20 rows

With the info sorting appropriately, we will write it to Amazon Easy Storage Service (Amazon S3) as a brand new desk within the AWS Glue Knowledge Catalog. We use the mapped DynamicFrame for this write as a result of we didn’t modify any knowledge previous that remodel:

# Create "demo" Database if none exists
spark.sql("create database if not exists demo")

# Set glueContext sink for writing new desk
s3output = glueContext.getSink(
s3output.setCatalogInfo(catalogDatabase="demo", catalogTableName="populations")

# Write out ‘mapped’ to a desk in Glue Catalog
s3output = glueContext.getSink(
s3output.setCatalogInfo(catalogDatabase="demo", catalogTableName="populations")

Lastly, we run a question towards our new desk to point out our desk created efficiently and validate our work:

choose * from demo.populations

Convert notebooks to AWS Glue jobs with nbconvert

Jupyter notebooks are saved as .ipynb recordsdata. AWS Glue doesn’t at present run .ipynb recordsdata straight, so that they must be transformed to Python scripts earlier than they are often uploaded to Amazon S3 as jobs. Use the jupyter nbconvert command from a terminal to transform the script.

  1. Open a brand new terminal or PowerShell tab or window.
  2. cd to the working listing the place your pocket book is.
    That is doubtless the identical listing the place you ran jupyter pocket book initially of this put up.
  3. Run the next bash command to transform the pocket book, offering the proper file identify in your pocket book:
    jupyter nbconvert --to script <Untitled-1>.ipynb

  4. Run cat <Untitled-1>.ipynb to view your new file.
  5. Add the .py file to Amazon S3 utilizing the next command, changing the bucket, path, and file identify as wanted:
    aws s3 cp <Untitled-1>.py s3://<bucket>/<path>/<>

  6. Create your AWS Glue job with the next command.

Word that the magics aren’t routinely transformed to job parameters when changing notebooks regionally. You have to put in your job arguments appropriately, or import your pocket book to AWS Glue Studio and full the next steps to maintain your magic settings.

aws glue create-job 
    --name is_blog_demo
    --role "<GlueServiceRole>" 
    --command {"Title": "glueetl", "PythonVersion": "3", "ScriptLocation": "s3://<bucket>/<path>/<"} 
    --default-arguments {"--enable-glue-datacatalog": "true"} 
    --number-of-workers 2 
    --worker-type G.2X

Run the job

After you’ve got authored the pocket book, transformed it to a Python file, uploaded it to Amazon S3, and eventually made it into an AWS Glue job, the one factor left to do is run it. Achieve this with the next terminal command:

aws glue start-job-run --job-name is_blog --region us-east-1


AWS Glue interactive periods provide a brand new strategy to work together with the AWS Glue serverless Spark atmosphere. Set it up in minutes, begin periods in seconds, and solely pay for what you employ. You should utilize interactive periods for AWS Glue job improvement, advert hoc knowledge integration and exploration, or for big queries and audits. AWS Glue interactive periods are usually out there in all Areas that help AWS Glue.

To be taught extra and get began utilizing AWS Glue Interactive Periods go to our developer information and start coding in seconds.

Concerning the creator

Zach Mitchell is a Sr. Massive Knowledge Architect. He works inside the product crew to boost understanding between product engineers and their prospects whereas guiding prospects via their journey to develop knowledge lakes and different knowledge options on AWS analytics companies.

Similar Posts

Leave a Reply

Your email address will not be published.