Evaluating large language models (LLMs) for text summarization is crucial for ensuring quality, reliability, and suitability for various applications. This session will demonstrate how to efficiently operationalize LLM evaluation using Amazon SageMaker and Bedrock, addressing challenges such as scalability, cost-effectiveness, and result centralization. Attendees will learn how to automate evaluation workflows, manage parallelization, and store results using a modular architecture that promotes flexibility and extensibility.

Agenda

Introduction to LLM Evaluation (5 minutes)
Importance and challenges of evaluating LLMs
Detailed explanation of the architecture using Amazon SageMaker and Bedrock
How to set up and trigger evaluation workflows
Live Demonstration (10 minutes)

Benefits of Attending

Understand the Importance of LLM Evaluation:Learn why evaluating LLMs is crucial for quality assurance, model improvement, ethical considerations, and user confidence.
Learn to Overcome Evaluation Challenges:Gain insights into addressing scalability, subjectivity, and interpretability challenges in LLM evaluation.
Automate and Scale Evaluations:Discover how to leverage AWS services to automate and scale LLM evaluation workflows efficiently.
Implement Cost-Effective Solutions:Learn about cost-effective strategies for evaluating LLMs using serverless architectures.
Hands-On Experience:Participate in a live demonstration and get practical experience in setting up and managing LLM evaluation processes.

Ayyanar Jeyakrishnan - AWS Community Builder Machine Learning serves

an Executive Director and Principal Engineer at Wells Fargo Bangalore, brings over 18 years of IT experience. Holding all 12 AWS certifications, Ayyanar specializes in crafting scalable and efficient multi-cloud solutions.

Successfully designing and implementing multi-cloud architectures for various organizations, Ayyanar emphasizes high availability, security, and scalability in Machine Learning and Data Platform projects. As a strong advocate for DevOps and MLOps methodologies. He provided Generative AI on AWS workshops, discussed in various technical events on AWS AI/ML Stack offerings and its business value and usecases.

Also he actively engages with the tech community, co-organizing the AWS User Group Bangalore. He shares his knowledge through his blog "dataopslabs" and YouTube videos, providing valuable technical insights.