ITTrainingNow

Large Language Models for Technical Teams: Prompting, Grounding, and Evaluation

Build reliable LLM features with solid prompt engineering, retrieval-augmented grounding, and repeatable evaluation.

Course overview

Learn to build LLM-powered features that are accurate, grounded, and measurable in this two-day introductory instructor-led course for technical teams. Understand how large language models process tokens and generate output, engineer prompts with system messages, examples, and structured output, ground responses in your own data with embeddings, vector search, and retrieval-augmented generation (RAG), connect models to live systems with tool calling, and evaluate LLM applications for quality, groundedness, safety, latency, and cost. Hands-on Python labs build and test a small document question-answering service from start to finish.

2 daysCourse length
8 modulesHands-on Python labs
IntroductoryNo LLM development experience required
Technical streamFor developers and IT professionals

Course coverage

Progress from LLM internals and prompt engineering to grounded applications with repeatable evaluation.

  • 01 LLM foundations: tokens, context windows, sampling, and model choice
  • 02 Calling LLM APIs: messages, parameters, streaming, keys, and cost
  • 03 Prompt engineering: system prompts, few-shot, reasoning, and JSON output
  • 04 Embeddings and vector search: chunking, indexing, and similarity
  • 05 Retrieval-augmented generation (RAG): grounding, citations, and hybrid search
  • 06 Tool calling, agents, and MCP: grounding in live data and systems
  • 07 Evaluating LLM apps: test sets, metrics, and LLM-as-a-judge
  • 08 Safety and operations: prompt injection, guardrails, monitoring, and cost

Participants will learn to

  • Explain tokens, context windows, temperature, and model trade-offs.
  • Call LLM APIs securely and manage tokens, latency, and cost.
  • Engineer and version prompts that return reliable structured output.
  • Create embeddings, chunk documents, and run vector searches.
  • Build a RAG pipeline that grounds answers and cites sources.
  • Connect models to tools and live data with function calling.
  • Evaluate groundedness, relevance, and accuracy with test datasets.
  • Defend against prompt injection and monitor LLM apps in production.
Practical outcome: Build a Python document question-answering service that grounds LLM answers in your own content with RAG and cited sources, add a tool call for live data, and create an automated evaluation suite that scores groundedness and accuracy so prompt and retrieval changes can be compared with evidence.

Two-day course outline

Day 1 - LLM foundations, APIs, and prompt engineering: Learn how large language models work, including tokens and tokenization, context windows, next-token prediction, temperature and top-p sampling, training versus inference, and why models hallucinate. Compare model families and providers including OpenAI, Anthropic, Google Gemini, and open models such as Llama and Mistral, and cloud platforms such as Azure AI Foundry, Amazon Bedrock, and Vertex AI, weighing quality, speed, context size, and cost. Call an LLM API from Python using chat messages, parameters, streaming, and secure API key handling, and track token usage. Engineer prompts with system prompts, roles, delimiters, few-shot examples, step-by-step reasoning, and structured JSON output validated against a schema. Manage prompt templates and versions in code. Labs: build a prompt-driven extraction and classification script and compare results across prompts and models.

Day 2 - Grounding with RAG and tools, evaluation, and safety: Ground LLM responses in your own data. Create embeddings, chunk and index documents, and run similarity search with a vector store. Build a retrieval-augmented generation (RAG) pipeline with context assembly, source citations, hybrid keyword and vector search, and reranking, and compare RAG with fine-tuning. Ground answers in live systems with tool calling and function calling, and introduce AI agents and the Model Context Protocol (MCP). Evaluate LLM applications with golden test datasets, metrics for groundedness, relevance, faithfulness, and accuracy, LLM-as-a-judge scoring, and evaluation tools such as promptfoo, Ragas, and Azure AI Foundry evaluations. Apply safety and operations practices: prompt injection, data leakage, guardrails, content filtering, logging and monitoring, latency, and cost control. Final lab: build, ground, and evaluate a document question-answering service and report improvements with evidence.

Format and audience

For: Developers, software engineers, data analysts, data engineers, IT professionals, and technical leads who want to build and evaluate LLM-powered features. Business staff should consider Large Language Models for Business.

Prerequisites: Basic programming experience, ideally in Python, and familiarity with JSON and calling web APIs. Generative AI Fundamentals for Technical Teams is helpful but not required. Participants need a computer with VS Code, Python 3, Git, and an LLM API key (instructor-provided or their own).

How it works: Two days of instructor-led explanations, demonstrations, and hands-on labs. Contact us to discuss dates and virtual or onsite delivery.

Explore the course

Request a quote for the two-day Large Language Models for Technical Teams course and discuss your team's training needs.

Request a Quote

Back to AI & ML training