Syntiant Corp., a leader in edge AI deployment, announced results of its first optimizations that reduce the computational footprint of leading large language model (LLM) architectures, enabling these massive neural networks to run on cloud-free, always-on devices at the edge of networks.
“We are focused on making edge AI a reality, bringing low power, highly accurate intelligence to edge devices,” said Kurt Busch, CEO at Syntiant. “Our core model optimizations enable considerable processing time acceleration across a range of compute platforms. Those same optimizations, which Syntiant has deployed across millions of devices worldwide, have been successfully applied to LLMs, bringing conversational speech to the edge, serving as the new interface between humans and machines.”
Syntiant used a novel algorithm to determine the sparsity fraction of LLMs to generate significant speedups in output token generation and reduce memory footprint. The company achieved 50% sparsity with minimal accuracy loss when computing with 8- bit quantized weights, significantly improving interpretability and processing power, while reducing cloud costs. These improvements, along with a custom SIMD (single instruction/multiple data) kernel and several other algorithmic innovations, enable Syntiant to achieve a 50% increase in output token generation speed on a LLaMa-7B benchmark.
“Syntiant continues to lead the category of low power NLP with production-ready solutions that directly address voice AI workload efficiency and could enable smarter conversational endpoints with generative AI. At M12, Microsoft’s Venture Fund, we’re proud to back Syntiant who we feel will help establish a ‘first mile’ standard for future speech-to-speech AI use cases,” said Michael Stewart, partner at M12.
Demoed live at CES 2024 in Las Vegas, Syntiant’s optimizations demonstrate an adjacent class of LLMs that run entirely at the edge. The ability to enable significant processing time acceleration brings meaningful benefits to end users from both latency and privacy perspectives, across a wide variety of consumer and commercial use cases, from earbuds to automobiles.
SOURCE: GlobeNewswire