Sony Pixel Power calrec Sony

Ray Shines with NVIDIA AI: Anyscale Collaboration to Help Developers Build, Tune, Train and Scale Production LLMs

18/09/2023

Large language model development is about to reach supersonic speed thanks to a collaboration between NVIDIA and Anyscale.

At its annual Ray Summit developers conference, Anyscale - the company behind the fast growing open-source unified compute framework for scalable computing - announced today that it is bringing NVIDIA AI to Ray open source and the Anyscale Platform. It will also be integrated into Anyscale Endpoints, a new service announced today that makes it easy for application developers to cost-effectively embed LLMs in their applications using the most popular open source models.

These integrations can dramatically speed generative AI development and efficiency while boosting security for production AI, from proprietary LLMs to open models such as Code Llama, Falcon, Llama 2, SDXL and more.

Developers will have the flexibility to deploy open-source NVIDIA software with Ray or opt for NVIDIA AI Enterprise software running on the Anyscale Platform for a fully supported and secure production deployment.

Ray and the Anyscale Platform are widely used by developers building advanced LLMs for generative AI applications capable of powering intelligent chatbots, coding copilots and powerful search and summarization tools.

NVIDIA and Anyscale Deliver Speed, Savings and Efficiency Generative AI applications are captivating the attention of businesses around the globe. Fine-tuning, augmenting and running LLMs requires significant investment and expertise. Together, NVIDIA and Anyscale can help reduce costs and complexity for generative AI development and deployment with a number of application integrations.

NVIDIA TensorRT-LLM, new open-source software announced last week, will support Anyscale offerings to supercharge LLM performance and efficiency to deliver cost savings. Also supported in the NVIDIA AI Enterprise software platform, Tensor-RT LLM automatically scales inference to run models in parallel over multiple GPUs, which can provide up to 8x higher performance when running on NVIDIA H100 Tensor Core GPUs, compared to prior-generation GPUs.

TensorRT-LLM automatically scales inference to run models in parallel over multiple GPUs and includes custom GPU kernels and optimizations for a wide range of popular LLM models. It also implements the new FP8 numerical format available in the NVIDIA H100 Tensor Core GPU Transformer Engine and offers an easy-to-use and customizable Python interface.

NVIDIA Triton Inference Server software supports inference across cloud, data center, edge and embedded devices on GPUs, CPUs and other processors. Its integration can enable Ray developers to boost efficiency when deploying AI models from multiple deep learning and machine learning frameworks, including TensorRT, TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS XGBoost and more.

With the NVIDIA NeMo framework, Ray users will be able to easily fine-tune and customize LLMs with business data, paving the way for LLMs that understand the unique offerings of individual businesses.

NeMo is an end-to-end, cloud-native framework to build, customize and deploy generative AI models anywhere. It features training and inferencing frameworks, guardrailing toolkits, data curation tools and pretrained models, offering enterprises an easy, cost-effective and fast way to adopt generative AI.

Options for Open-Source or Fully Supported Production AI Ray open source and the Anyscale Platform enable developers to effortlessly move from open source to deploying production AI at scale in the cloud.

The Anyscale Platform provides fully managed, enterprise-ready unified computing that makes it easy to build, deploy and manage scalable AI and Python applications using Ray, helping customers bring AI products to market faster at significantly lower cost.

Whether developers use Ray open source or the supported Anyscale Platform, Anyscale's core functionality helps them easily orchestrate LLM workloads. The NVIDIA AI integration can help developers build, train, tune and scale AI with even greater efficiency.

Ray and the Anyscale Platform run on accelerated computing from leading clouds, with the option to run on hybrid or multi-cloud computing. This helps developers easily scale up as they need more computing to power a successful LLM deployment.

The collaboration will also enable developers to begin building models on their workstations through NVIDIA AI Workbench and scale them easily across hybrid or multi-cloud accelerated computing once it's time to move to production.

NVIDIA AI integrations with Anyscale are in development and expected to be available by the end of the year.

Developers can sign up to get the latest news on this integration as well as a free 90-day evaluation of NVIDIA AI Enterprise.

To learn more, attend the Ray Summit in San Francisco this week or watch the demo video below.

See this notice regarding NVIDIA's software roadmap.
LINK: https://blogs.nvidia.com/blog/2023/09/18/llm-anyscale-nvaie/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

02/05/2026

Dalet Flex LTS Delivers Smarter Search, Faster Editing, and an AI-Ready Foundation for Modern Media

Dalet, a leading technology and service provider for media-rich organizations, t...

01/05/2026

NBCUniversal's Peacock to Be First Streamer to Integrate Dolby's Full Suite of Premium Picture and Sound Innovations

January 5 2026, 18:30 (PST) NBCUniversal's Peacock to Be First Streamer to ...

01/04/2026

DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION

January 4 2026, 18:00 (PST) DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION Douyin Users Can Now Create And Share Videos With Stun...

27/02/2026

Atomos Introduces Ninja RAW HDR Monitor-Recorder At CP+ Trade Show

Share Copy link Facebook X Linkedin Bluesky Email...

27/02/2026

Scripps to Reacquire 23 ION Stations

Share Copy link Facebook X Linkedin Bluesky Email...

27/02/2026

Survey: 88% of Sports Execs Bullish on Industry's Prospects

Share Copy link Facebook X Linkedin Bluesky Email...

27/02/2026

Chyron Releases Weather 2.3 With New Data and Automation Features

Share Copy link Facebook X Linkedin Bluesky Email...

27/02/2026

Samba: Essentials, Utilities Tally Most Ad Impressions in 2nd-Half 2025

Share Copy link Facebook X Linkedin Bluesky Email...

27/02/2026

TVB: Linear TV Still King

Share Copy link Facebook X Linkedin Bluesky Email...

27/02/2026

AJA U-TAP Helps Stary Technologies with Courtroom Evidenc...

Video is one of the lawyer's most powerful storytelling tools in civil litigation today, whether used to transport jurors to an incident scene or challenge ...

27/02/2026

Foundry releases Nuke 17-0

Creative software developer Foundry today released Nuke 17.0, the latest version of its powerful compositing tool for visual effects and animation. Marking one ...

27/02/2026

KRK helps Kryz Reid Keep Third Eye Blind Sounding Sharp O...

In a sun-drenched Los Angeles studio filled with guitars, laughter, and the low thrum of KRK monitors, Third Eye Blind's Kryz Reid balances rockstar energy ...

27/02/2026

Atomos Announces Ninja RAW Monitor-Recorder

Atomos announces the launch of Ninja RAW, a 5-inch HDR monitor-recorder designed to give filmmakers, content creators, and broadcasters an uncompromising and af...

27/02/2026

Marshall Electronics CV228 Lipstick Cameras Capture Criti...

FM, leaders in commercial property insurance, is utilizing Marshall Electronics CV228 Weatherproof Lipstick Cameras to capture critical fire testing data inside...

27/02/2026

Telestream Advances Production-Ready AI Across Its Produc...

Telestream, global leader in media workflow technologies, today announced expanded practical AI enhancements across its Vantage, Vantage Cloud, EDC, Stanza, and...

27/02/2026

Teatro alla Scala Elevates Backstage Communication with R...

Riedel Communications today announced that Fondazione Teatro alla Scala has deployed a comprehensive wireless intercom solution, leveraging Riedel's Bolero,...

27/02/2026

Screen Australia announces Rachel Perkins as Director of First Nations Strategy

27 02 2026 - Media release Screen Australia announces Rachel Perkins as Director of First Nations Strategy Rachel Perkins. Photo credit: John Platt. Screen ...

27/02/2026

RT Media Sales: SIRO to sponsor Home of the Year on RT One

One of Ireland's favourite lifestyle shows Home of the Year, returns for its 12th series and will be proudly sponsored by SIRO. The brand-new series will ai...

27/02/2026

Our Farm: A GIY Story lands on RT One and RT Player from March 3

Our Farm: A GIY Story lands on RT One and RT Player from March 3 From walled garden to community farm, new six-part series captures the unfiltered realities...

27/02/2026

Seachtain na Gaeilge 2026 ar RT

Note: English version included below the Irish language version. Seachtain na Gaeilge 2026 ar RT A Ghaeilge mo cheol th ' T feachtas athnuaite i n...

26/02/2026

NHL, Cosm Install C360 10.5K Capture Systems at All 32 Arenas

Multi-angle coverage, on-demand access to ultra-high-resolution video are provided for replays and clips across multiple distribution channels The NHL and Cosm...

26/02/2026

Advanced Systems Group Promotes Jody Boatwright to Chief Strategy Officer

Advanced Systems Group, LLC (ASG), a technology and services provider for media creatives and content owners, has appointed Jody Boatwright as Chief Strategy Of...

26/02/2026

TikTok, MLB Expand Multi-Year Global Content Partnership

As players report to Spring Training, TikTok and MLB announce an expanded content partnership bringing baseball fans around the world closer to the game through...

26/02/2026

Chyron Announces 2026 Designer of the Year Competition

Chyron announces its 2026 Designer of the Year Competition to be awarded during a live stream from the 2026 NAB Show in Las Vegas. This year's competition i...

26/02/2026

Appear's X Platform Now SRT Verified for YouTube Live Content Delivery

Appear, which specializes in live production technology, announces that its X Platform has been officially verified by YouTube for Secure Reliable Transport (SR...

26/02/2026

Adder Technology Enters New Era of Strategic Engagement with Launch of Customer Experience Center

Adder Technology, a specialist in connectivity solutions and high performance IP...

26/02/2026

Alcom Elevates Headend Video Service with Harmonic to Drive Growth

Harmonic announces that Alcom, a leading telco operator in Finland, is powering its next-generation white-label headend video service with Harmonic's XOS Ad...

26/02/2026

ESPN, Disney+ to Deliver More Banana Ball Games Than Ever as Collaboration Expands in 2026

ESPN, Disney , and the Savannah Bananas announce a 25-game exclusive package in ...

26/02/2026

SailGP Announces Broadcast Extensions with Globo and BandSports in Brazil for 2026 Season

Brazilian broadcaster Globo has returned as the official broadcast partner of th...

26/02/2026

NDI Appoints JRS as Official Distribution Partner for China

NDI, which concentrates in plug-and-play IP video connectivity, announces a strategic partnership with Jiaruisen (JRS), a Shenzhen-based technology distributor,...

26/02/2026

SVG Students To Watch: Noah Boudolf, Clemson University

The Charleston, SC, native has excelled as a live-camera operator on Tigers broadcasts In the live-sports-video industry, the future is bright. Our series SVG ...

26/02/2026

Behind The Mic: ESPN, Tennis Channel Sign New Deals for 2026 Tennis Analysts

Behind The Mic provides a roundup of recent news regarding on-air talent, including new deals, departures, and assignments compiled from press releases and repo...

26/02/2026

SVG in Indy: Beyond the Colts, Lucas Oil Stadium Stays Busy With NFL Scouting Combine, NCAA Men's Final Four

Supported by partners, the in-venue production team keeps videoboard-show qualit...

26/02/2026

SVG New Sponsor Spotlight: CAMB.AI's Akshat Prakash on AI Translation and the Future of Live Sports Localization

CAMB.AI provides AI speech synthesis and translation with a focus on localizatio...

26/02/2026

SVG Sit-Down: AWS's Regina Rossi on the Company's New Vertical-Video-Cropping Tool

AWS Elemental Inference could help sports broadcasters learn to love AI This we...

26/02/2026

Rohde & Schwarz awarded contract by Israel Airports Authority for QPS201 security scanners

Rohde & Schwarz awarded contract by Israel Airports Authority for QPS201 securit...

26/02/2026

Rohde & Schwarz highlights its unique CMX500 one-box tester tailored for NTN testing at MWC Barcelona 2026

Rohde & Schwarz highlights its unique CMX500 one-box tester tailored for NTN tes...

26/02/2026

Rohde & Schwarz high-efficiency transmitter powers next-gen broadcast services in Serbia

Rohde & Schwarz high-efficiency transmitter powers next-gen broadcast services i...

26/02/2026

Rohde & Schwarz highlights its comprehensive embedded systems test solutions at embedded world 2026

Rohde & Schwarz highlights its comprehensive embedded systems test solutions at ...

26/02/2026

Rohde & Schwarz to showcase spectrum security and network efficiency solutions at Mobile World Congress 2026

Rohde & Schwarz to showcase spectrum security and network efficiency solutions a...

26/02/2026

Rohde & Schwarz and Broadcom showcase first Wi-Fi 8 RF signaling tests, paving way for next-gen connectivity

Rohde & Schwarz and Broadcom showcase first Wi-Fi 8 RF signaling tests, paving w...

26/02/2026

Rohde & Schwarz advances AI-RAN testing using digital twins with NVIDIA

Rohde & Schwarz advances AI-RAN testing using digital twins with NVIDIA Rohde & Schwarz, in collaboration with NVIDIA, continues to drive AI-RAN innovation fo...

26/02/2026

Rohde & Schwarz and LITEON demonstrate highthroughput 5G femtocell testing with the PVT360A

Rohde & Schwarz and LITEON demonstrate high throughput 5G femtocell testing with...

26/02/2026

Cobalt Digital Returns to ISE with Comprehensive Lineup of IPMX-Ready Solutions

Industry's only IPMX-compliant audio monitor takes centerstage along with IPMX multiviewers, gateways, converters and routers Champaign, IL - January 26, 2...

26/02/2026

Cobalt Digital Products Receive IPMX Product Certification Following Inaugural Testing Event

Cobalt among first manufacturers to achieve verified compliance status with IPMX...

26/02/2026

COBALT ARIA AUD-MON Receives Best of Show Awards at ISE 2026

Barcelona - 20 February 2026 Cobalt Digital, the leading designer and manufacturer of award-winning ST 2110 and SDI signal processing products, and a founding...

26/02/2026

SES Extends EGNOS GEO-1 Satellite Service to Power Precise Navigation Across Europe

The agreement ensures Europe's satellite-based augmentation continues enhanc...

26/02/2026

AWS Launches New Tool for Vertical Video Conversion

Share Copy link Facebook X Linkedin Bluesky Email...