India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
LLM and AI Platform Selection Based on Your Own Tasks and Data
Model and platform selection is a structured evaluation that decides which language models, and which cloud AI platform, an organisation should use for each of its GenAI workloads, based on measured quality on its own tasks, cost per completed task, latency, data residency and licensing. NTSPL runs these bake-offs for CIOs choosing between AWS, Azure and Google Cloud or among models.
Delivered on
In scope
Why Model Choice Must Be Measured per Task, Not Read from Leaderboards
Model choices are usually made by habit or by headline. One team picked a model because it topped a public leaderboard; another because the cloud account already existed. Six months later the organisation pays frontier-model prices to classify emails a small model could handle, while a Marathi-language use case performs poorly because nobody tested it in Marathi. Public benchmarks measure general knowledge and coding puzzles, not whether a model can extract a BOQ line from a scanned tender or tag a 1980s news clip correctly.
NTSPL replaces opinion with a bake-off on your tasks. We build an evaluation set of 150-400 real items per use case, define scoring rubrics with domain experts, and run 4-8 candidate models across the platforms you are willing to use. Every candidate is scored on task quality, cost per successful task at expected volume, p95 latency, availability in Indian regions and licence terms. The output is a model portfolio and routing policy, not a single winner, plus a platform recommendation with switching costs stated.
When a Formal Model and Platform Bake-off Is Worth Running
Signals you need Model & Platform Selection
- Three or more GenAI use cases are planned and each team is choosing models independently.
- Monthly model spend has crossed ₹10 lakh and nobody has tested whether cheaper models meet the quality bar.
- Leadership must decide between AWS, Azure and Google Cloud for an AI platform, or justify using more than one.
- Use cases involve Indian languages, scanned documents or domain jargon where public benchmarks say little.
- A regulator, board or customer contract requires documented reasons for model choice and data location.
When to choose something else
- If there is a single, low-volume use case already working well on an acceptable model, a formal bake-off adds little; monitor quality and cost instead.
- If the use case is not yet defined, start with use-case prioritisation; models cannot be compared against undefined tasks.
- If the platform is fixed by an enterprise agreement and only one or two models are permitted, run a lighter evaluation within that boundary.
What the Selection Engagement Evaluates and Decides
Use-case task definitions
Precise task statements, input types, output formats and success criteria for each use case, agreed with business owners.
Evaluation datasets
150-400 real items per task with expert reference outputs, stratified by difficulty, language and document quality.
Scoring rubrics and judges
Exact-match, field-level and rubric scoring, with LLM-as-judge calibrated against human graders on a sample.
Candidate shortlist
4-8 models across proprietary and open-weight families, filtered first by residency, licence and platform availability.
Cost and latency modelling
Cost per successful task at projected volume, including retries, caching and batch options, plus p50 and p95 latency.
Platform assessment
Comparison of governance, identity, networking, regional availability, enterprise agreement terms and skills on each cloud.
Task-Specific Evaluation Harness for Comparing Models and Platforms
Task-Specific Model Bake-off and Evaluation Harness
Curated task datasets run through the same prompts and pipelines against each candidate model on each platform; outputs are scored by rule-based checks, calibrated LLM judges and human graders. Scores combine with cost and latency measurements into a weighted decision matrix and routing policy.
Select any component
Task-Specific Model Bake-off and Evaluation Harness: click any of its 9 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Comparing Bedrock, Microsoft Foundry and Gemini Enterprise Agent Platform
Bedrock gives one API across Amazon Nova, Anthropic, Meta, Mistral and other models, with built-in automatic, human and LLM-as-judge evaluation jobs; open-weight models can be imported or hosted on SageMaker AI.
Foundry Models offers OpenAI, Microsoft, Meta, Mistral and other families under one governance model, with evaluation runs and model comparison inside the same project.
Model Garden exposes Gemini, Gemma, partner and open models; the Gen AI evaluation service runs pointwise and pairwise metrics, and open models can be served on managed endpoints or Cloud Run with GPUs.
Frontier vs Small, Proprietary vs Open-Weight, One Cloud vs Several
Frontier model versus small model
For extraction, classification and short-form drafting, small and mid-sized models often score within 2-4 points of frontier models at a fifth to a twentieth of the cost per task. Reserve frontier models for multi-step reasoning, long-document synthesis and agent planning. If a small model is within your quality tolerance on the hold-out set, it should be the default, with escalation to a larger model on low confidence.
Proprietary versus open-weight models
Open-weight models suit high, steady volumes where self-hosting on reserved GPUs beats per-token pricing, strict residency needs and fine-tuning for narrow tasks. They add operational burden: patching, scaling and safety tuning become yours. Check licence terms, including usage thresholds and acceptable-use clauses, before shortlisting. Below roughly ₹5-8 lakh a month in model spend, managed proprietary models are usually cheaper all-in.
One cloud platform or several
Standardise on one primary AI platform where your identity, data and enterprise agreements already sit; this simplifies governance, networking and skills. Add a second platform only when a specific model with a measured, material quality advantage is unavailable on the primary, and access it through the same AI gateway. We document switching costs so the choice is reversible.
Weighting the decision
Weights should differ by use case. A customer-facing assistant might weight quality 45%, latency 20%, cost 20% and residency 15%; a nightly batch summariser might weight cost 50%. Agree weights before seeing results to avoid rationalising a favourite. A model that fails any hard constraint, such as in-India processing for personal data, is excluded regardless of score.
Four-to-Six-Week Bake-off Method
Use-case framing and constraints
Define tasks, volumes, latency needs, residency and licence constraints, and agree decision weights with the steering group.
Dataset and rubric build
Collect real items, write reference outputs with experts and design scorers; seal a 30% hold-out.
Candidate runs
Run 4-8 models across platforms with fair per-model prompts, recording quality, tokens, latency and errors.
Judge calibration and blind review
Validate LLM judges against human grades and run blind expert review on a sample of each model's outputs.
Decision and routing policy
Produce the decision matrix, model portfolio, routing rules, platform recommendation and re-evaluation triggers.
What you receive
Scores That Drive the Selection Decision
| KPI | Typical target |
|---|---|
| Task quality score on sealed hold-out set | Meets or exceeds agreed threshold per use case |
| Cost per successful task at projected volume | Typically 30-70% below initial default choice |
| p95 latency per task | Within the experience budget set for each use case |
| LLM judge agreement with human graders | Cohen's kappa of 0.6 or higher before use |
| Use cases with a documented model rationale | 100% |
Selection Mistakes That Cost Money Later
| Risk | How NTSPL prevents it |
|---|---|
| Bake-off results overfit to the prompts used | Per-model prompt tuning on the development set only, with final scores reported on a sealed hold-out set. |
| LLM judges favour outputs from their own model family | Judges from a different family than candidates where possible, calibration against human grades and blind expert sampling. |
| Chosen model is retired or repriced within months | Routing through an AI gateway, reusable evaluation sets and a documented re-evaluation trigger. |
| Residency or licence issues discovered after build | Hard-constraint filter applied before shortlisting, with legal review of open-weight licence terms. |
Amazon Bedrock vs Microsoft Foundry vs Google's Gemini Enterprise Agent Platform: which AI platform should we choose?
Choose the platform where your identity, data and commercial commitments already sit, unless a measured quality gap on your tasks says otherwise. Organisations on Microsoft 365, Entra ID and Fabric usually gain most from Foundry and its OpenAI models. Those with data lakes on AWS benefit from Bedrock's breadth of model families under one API. BigQuery-centred organisations find Gemini Enterprise Agent Platform simplest, with strong multimodal and long-context models. All three offer enterprise controls and Indian regions for core services. Decide with a bake-off on your own data, then route any exceptional model needs through a common gateway rather than splitting governance.
Key facts: Model & Platform Selection
Model & Platform Selection in Practice
2 Model & Platform Selection scenarios, from Engineering, procurement and construction (EPC tendering) and Media (broadcast news archive), plus related platform case studies.
EPC contractor picks a three-model mix for tender analysis and cuts projected cost 58%
Media archive chooses an open-weight model for Bengali and Hindi clip tagging at scale
Wealth firm's research assistant cuts RM client-prep time from 50 to 12 minutes
Infrastructure company's legal team reviews contracts in hours, not days
Model & Platform Selection: Frequently Asked Questions
Why not just pick the top model on public leaderboards?
Leaderboards test general reasoning, coding and knowledge, not your documents, languages or formats. We regularly see a mid-ranked model outperform a leader on a specific extraction or Indian-language task, and at a fraction of the cost. Only an evaluation on your own tasks tells you which model is good enough.
How long does a model bake-off take and what does it cost?
For two or three use cases, a bake-off typically takes four to six weeks with indicative NTSPL fees of ₹8-18 lakh. Model usage during evaluation is usually ₹50,000 to ₹3 lakh. The datasets and harness remain yours, so later re-evaluations take days rather than weeks.
Which models are available in Indian cloud regions?
Availability changes frequently and differs by provider, model and deployment type. Each platform offers some models in its Indian regions, while others are only reachable through global or cross-region endpoints that may process data elsewhere. We check current availability for each candidate during shortlisting and treat in-India processing as a hard constraint where personal data is involved.
Should we fine-tune a model instead of choosing a bigger one?
Sometimes. Fine-tuning a small model on a few thousand high-quality examples can match a larger model on a narrow, stable task at much lower serving cost. It adds training, versioning and re-evaluation work. We include a fine-tuned candidate in the bake-off only when volume is high and the task is stable.
How do Indian-language tasks change model selection?
Considerably. Quality in Hindi, Tamil, Bengali or Marathi varies more between models than English quality does, and tokenisation can make Indic text several times more expensive per word. We test each language separately and include models trained with stronger Indic coverage in the shortlist.
How often should we re-run the evaluation?
Re-run it when a model you use is announced for retirement, when a new candidate offers a likely cost or quality gain, or every six months for high-spend use cases. Because the datasets and harness already exist, a re-run usually takes three to five days.