AWS Glue is Amazon’s service for serverless data integration, and their latest version, 5.1, improves the performance of engines, increases security, and adds open-table formats to their supported formats.
AWS Glue simplifies data management by finding, preparing, moving, and combining data from disparate sources. This allows customers to derive more insights from their data for analytics, machine learning, and application development. With this release, AWS Glue upgrades its core engines to Apache Spark 3.5.6, Python 3.11, and Scala 2.12.18 for enhanced performance and security.
This release adds support for the open-table format libraries. These include Apache Hudi 1.0.2, Apache Iceberg 1.10.0, and Delta Lake 3.3.2. AWS Glue 5.1 now supports Iceberg format version 3.0. You will get default column values, deletion vectors for merge-on-read tables, multi-argument transforms, and row lineage tracking with this update.
Also Read: Snowflake Partners with Select Star for AI-Powered Data Discovery
AWS Glue 5.1 improves the write operation access control in AWS Lake Formation by adding support for DML and DDL operations for Spark DataFrames and Spark SQL. Previously, access control was only applicable to a read operation. With this release, Apache Spark now fully has table access control for Hudi and Delta Lake tables. This provides more security and governance options for customers on their data lakes.
AWS Glue 5.1 is now available in multiple regions worldwide, such as US East (N. Virginia, Ohio); US West (Oregon); Europe (Ireland, Stockholm, Frankfurt, Spain); Asia Pacific: Hong Kong, Singapore, Sydney, Tokyo, Malaysia, Thailand, Mumbai; and South America: São Paulo.
AWS allows companies to better integrate and manage large volumes of data. This further enhances data governance, security, and flexibility in modern data settings.