Exclusive: Zoho backs smaller enterprise LLMs for AI future

Zoho has declared it is taking a different path in artificial intelligence, unveiling its own smaller, enterprise-grade large language models and rejecting the trend for ever-expanding general-purpose systems.

"We don't want it to be a general purpose LLM," said Ramprakash Ramamoorthy, Zoho's Director of AI Research, during an interview at Zoholics Sydney 2025.

"We're never going to expose it via a chatbot where you can, I mean, so it's not ChatGPT, it's not that. This is purely enterprise grade. It's a part of our AI infrastructure. This will be powering AI features across Zoho platform."

The company has launched three models with 1.3, 2.6 and 7 billion parameters, compared with industry standards of 32 or 70 billion. "These are much smaller because we know in the enterprise sense, there is a lot of context. And with smaller models, I get less compute footprint, less data that is required to make these models work," he explained.

"A 7 billion model is generative enough. I mean, it can generate things that it has not seen before, but at the same time, it's bound to the context."

He drew a clear distinction with rivals who rely on massive models for every problem.

"One is companies that make everything a large language model problem. You don't need a large language model to find out an anomaly. You don't need a large language model to find out the best time to contact your customer. Now that is very data inefficient, compute inefficient," he said.

"By right sizing models, we have an array of models. One, there are purpose built models for sentiment, for finding the time to contact your customers. And these models do only one thing. Then there is an LLM which can do 10 things."

The decision reflects Zoho's long-running strategy of building its own infrastructure. "The way we see this going is Zoho plays the long game. And interestingly, you see all these typical enterprise SaaS companies, they pay a huge chunk of their profits to data centre providers today. Increasingly, they are paying a huge chunk of their profits to model service providers, and we don't want to do that," he said. "We took that decision 15 years back where we built our own data centres, and today we know that's part of the value that we offer to customers. Same thing will happen to model service providers as well."

Privacy is another driving force. "We never know what the model service provider could be using the data for. So these are the two prime reasons. One is cost, and I mean cost in terms of compute. And then two is the privacy cost. So these are the reasons we launched our own LLM," he said.

Alongside its models, Zoho has launched Zia agents, an Agent Studio and a marketplace.

"These LLMs are pretty siloed in the sense before agents, an LLM is like a PC that's not connected to anything. But then agents are the internet connection to these LLMs. They can look up multiple systems, get you the data that you want," he explained. "We understand that these agents are not one size fits all. So the LLM is going to be horizontal, but the agents are going to be very vertical specific. So an agent for banks in Australia, an agent for construction industry in New South Wales. So it could get very specific."

On adoption in the region, Ramamoorthy described two clear patterns.

"Once you ask this question, I had the realisation the adoption is very good at the lower end of the spectrum, like less than 10 users, the adoption is very good. In the enterprise space, it's the middle market that is sluggish," he said. "Small businesses like solopreneurs are getting a lot of benefits because they have an L1 agent taking care of solving their customer support L1 customer support issue. An enterprise is reaping a lot of benefits because an enterprise has everything scattered across systems. Now you have an agent to collect that information for you."

He added that the level of digital maturity also determines AI use. "The more digitised sectors are using more AI because digital maturity directly translates to AI maturity, and the rest are catching up," he said.

Banking and finance are leading the way. "We see a lot of adoption in the banking and finance sectors especially," he said.

"Somebody applies for a loan and then you ask for a bunch of documents. Now to, for a human to verify that documents, it's going to take time. Now, can you use an AI agent to verify these documents? In fact, that is the fastest, where we are seeing growth. The slowest is going to be healthcare. Because, again, there is a lot of privacy. There is a lot of the regulator pushes for you need more of the human."

Building the models has brought technical challenges. "It's not about training the model, it's about setting the high performance computing setup is where we had a lot of learnings," he said. "Last year we had a partnership. We announced a partnership with Nvidia, where Nvidia really helped us set up the infrastructure. It was an engineering partnership, and we've spent $20 million over the last year with Nvidia. So this partnership has enabled that. The bigger challenge was high performance computing and not AI model training."

He acknowledged the ongoing supply issues. "Even now, there is this geographical limits on the GPUs that you can import. So we have things in the US data centre, but the other data centres are getting it really slow because you are choked by the GPU supply. So all the issues, all the learnings that we had, is related to hardware," he said.

Zoho is measuring success in practical terms.

"We deployed AI in Zoho Legal Help Desk. Increasingly, as we move up market, our Legal Help Desk is overloaded, so we deployed AI, and we were able to see a 20 to 30 per cent improvement in productivity, nothing more than that," he said. "If I'm a support person answering 10 tickets a day. Now, if we can answer 12 tickets a day, that's a good productivity improvement."

For the future, Zoho sees AI as one element in a broader journey. "We want Zoho AI to enable the next best action at any given point in time," he said. "But I strongly believe this is something like digital transformation, where we've been talking about digital transformation for 20 years now, it's still an ongoing process. Things keep changing. All of a sudden there is mobile apps, all of a sudden there is some high speed internet. Then all of a sudden there is an LLM. So I think it's going to be a journey. You cannot do like five things and say, I've achieved AI."

He was emphatic about Zoho's strategy. "We don't want to give into the hype as well. We don't want to rebrand ourselves as Zoho.ai company. No, we strongly believe AI is one of the pieces of the puzzle, like enterprise search, like single sign on."

"AI will be one integral aspect of enterprise software," he said.