Cloudian announced the release of a new open-source software contribution that integrates PyTorch, the popular machine learning (ML) library, with local data lakes running on Cloudian HyperStore S3-compatible object storage. This breakthrough simplifies the machine learning workflow and reduces costs by allowing data scientists and AI developers to run ML on data resident in local Cloudian object storage, without the need to move and stage the data into another system. The ML tasks can also run on local compute resources such as AWS Outposts and Local Zones.
AWS Outposts and Local Zones users can now employ Python and machine learning libraries to analyze data within a local Cloudian HyperStore S3-compatible storage system without the cumbersome step of moving data to a separate staging area, streamlining the data processing pipeline and significantly accelerating the machine learning workflow. Cloudian is a certified Service Ready partner for AWS Outposts and Local Zones, and is commercially available through the AWS Marketplace.
This open-source contribution bridges the gap between distributed S3-compatible object storage systems and machine learning compute platforms, eliminating the dependency on a dedicated parallel file system for machine learning workflows. By enabling direct access to a cost-effective, scalable data repository, Cloudian is simplifying the machine learning process, reducing both complexity and costs associated with data analysis.
Also Read: Lenovo Accelerates Telco Transformation with Next-Generation Edge AI Innovations at MWC ‘24
Key Benefits of this development include:
- Simplified Workflow: Eliminates the need for data staging, thus simplifying the workflow and reducing the cost of real-time analysis and model training.
- Seamless Integration: Allows direct use of PyTorch with Cloudian HyperStore, enabling local S3-compatible data storage.
- Local Performance: Run machine learning models locally with AWS Outposts and Local Zones for low latency and high-speed access to data.
“We are excited to offer the machine learning community a tool that integrates two of their most important needs: the computational power of PyTorch and the storage flexibility of Cloudian S3-compatible systems,” said Jon Toor, Chief Marketing Officer of Cloudian. “By connecting these platforms, we are enabling a more efficient and streamlined approach to machine learning.”
Cloudian contributed enhancements to AWS Labs’ open-source S3-Connector-for-PyTorch. The enhancements enable PyTorch ML algorithms to access data in Cloudian’s HyperStore object storage system via the AWS S3 API. The enhanced S3 connector is available from the GitHub repositories of AWS Labs and Cloudian.
SOURCE: GlobeNewswire