Skip to content

Update Glue executor defaults #395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
remotesynth wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update Glue executor defaults #395

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/content/docs/aws/capabilities/config/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ Also see [OpenSearch configuration variables](#opensearch) which are used to man

| Variable | Example Values | Description |
| - | - | - |
| `GLUE_JOB_EXECUTOR_PROVIDER` | `v1` (default) \| `v2` | Feature flag to switch between the old and new Glue job executor providers. |
| `GLUE_JOB_EXECUTOR_PROVIDER` (**deprecated**) | `v1` \| `v2` (default) | Feature flag to switch between the old and new, default Glue job executor providers. |
| `GLUE_JOB_EXECUTOR` | `docker` (default) \| `kubernetes` | Whether to run Glue jobs when LocalStack is deployed on Kubernetes. Jobs are run as pods in the Kubernetes cluster. |
| `DOCKER_GLOBAL_IMAGE_PREFIX` | | Specify custom images for Glue jobs by configuring their custom image repository. |

Expand Down
20 changes: 3 additions & 17 deletions src/content/docs/aws/services/glue.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: Glue
description: Get started with Glue on LocalStack
tags: ["Ultimate"]
tags: ['Ultimate']
---

import FeatureCoverage from "../../../../components/feature-coverage/FeatureCoverage";
import FeatureCoverage from '../../../../components/feature-coverage/FeatureCoverage';

## Introduction

Expand All @@ -14,21 +14,7 @@ LocalStack allows you to use the Glue APIs in your local environment.
The supported APIs are available on our [API Coverage section](#api-coverage), which provides information on the extent of Glue's integration with LocalStack.

:::note
LocalStack now includes a container-based Glue Job executor, enabling Glue jobs to run within a Docker environment.
Previously, LocalStack relied on a pre-packaged binary that included Spark and other required components.
The new executor leverages the `aws-glue-libs` Docker image, provides better production parity, faster startup times, and more reliable execution.

Key enhancements include:

- Running Glue jobs inside Docker containers
- Providing isolated execution environments per job
- Executing multiple jobs in parallel
- Ensuring correct versioning of Spark, Hadoop, Python, Java, and related libraries
- Improving startup times and offline execution support
- Ability to run Glue jobs on Kubernetes clusters

To use it, set `GLUE_JOB_EXECUTOR=docker` and `GLUE_JOB_EXECUTOR_PROVIDER=v2` in your LocalStack configuration.
The new executor additionally deprecates older versions of Glue (`0.9`, `1.0`, `2.0`).
As of LocalStack 4.13.0, the default Glue job executor provider is `v2`, meaining LocalStack will use the container-based Glue Job executor, enabling Glue jobs to run within a Docker environment. The `GLUE_JOB_EXECUTOR_PROVIDER` configuration variable that enables the use of the legacy Glue job executor provider is deprecated and will be removed in the next major release.
:::

## Getting started
Expand Down