Pretrain Vision and Large Language Models in Python (Emily Webber Andrea Olgiati)（Z-Library）
Python
Master the art of training vision and large language models with conceptual fundaments and industry-expert guidance. Learn about AWS services and design patterns, with relevant coding examples Key Features Learn to develop, train, tune, and apply foundation models with optimized end-to-end pipelines Explore large-scale distributed training for models and datasets with AWS and SageMaker examples Evaluate, deploy, and operationalize your custom models with bias detection and pipeline monitoring Book Description Foundation models have forever changed machine learning. From BERT to ChatGPT, CLIP to Stable Diffusion, when billions of parameters are combined with large datasets and hundreds to thousands of GPUs, the result is nothing short of record-breaking. The recommendations, advice, and code samples in this book will help you pretrain and fine-tune your own foundation models from scratch on AWS and Amazon SageMaker, while applying them to hundreds of use cases across your organization. With advice from seasoned AWS and machine learning expert Emily Webber, this book helps you learn everything you need to go from project ideation to dataset preparation, training, evaluation, and deployment for large language, vision, and multimodal models. With step-by-step explanations of essential concepts and practical examples, you'll go from mastering the concept of pretraining to preparing your dataset and model, configuring your environment, training, fine-tuning, evaluating, deploying, and optimizing your foundation models. You will learn how to apply the scaling laws to distributing your model and dataset over multiple GPUs, remove bias, achieve high throughput, and build deployment pipelines. By the end of this book, you'll be well equipped to embark on your own project to pretrain and fine-tune the foundation models of the future. What you will learn Find the right use cases and datasets for pretraining and fine-tuning Prepare for large-scale training with custom accel
5
Views
Registered users can read the full content for free
Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.
Page
1
(This page has no text content)
Page
2
BIRMINGHAM—MUMBAI Pretrain Vision and Large Language Models in Python Copyright © 2023 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Group Product Manager: Ali Abidi
Page
3
Publishing Product Manager: Dhruv Jagdish Kataria Content Development Editor: Priyanka Soam Technical Editor: Rahul Limbachiya Copy Editor: Safis Editing Project Coordinator: Farheen Fathima Proofreader: Safis Editing Indexer: Subalakshmi Govindhan Production Designer: Ponraj Dhandapani Marketing Coordinator: Shifa Ansari and Vinishka Kalra First published: May 2023 Production reference: 1250523 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-80461-825-7 www.packtpub.com
Page
4
To the beginner's mind in us all. May we all have the humility and the courage to face the unknown with curious minds.
Page
5
- Emily Webber Foreword Welcome to the remarkable world of Machine Learning and foundation models! It has been rare that a new technology has taken the world by storm like these models have. These extraordinary creations have revolutionized the way we interact with technology and have opened up unprecedented opportunities for innovation and discovery. In this book, penned with clarity and a genuine passion for the subject, Emily invites you on a journey. Whether you are a first-time user or someone seeking a fresh perspective on these fascinating tools and practical applications, you will find this book a valuable companion. The field of ML can be daunting - complex algorithms, intricate mathematical formulas, and technical jargon. Yet Emily skillfully navigates through this intricacy, distilling the fundamentals into an accessible and easy-to-understand language. Emily’s writing style takes you not only through the "how" but also the "why" behind these advancements. The clear examples, and the grounding in the very practical platform offered by Amazon SageMaker, offer you the ability to follow along and learn-by- doing. The broad coverage of fields such as dataset preparation, pretraining, fine-tuning, deployment, bias detection and ML operations are covered in a rare soup-to-nuts, deep-yet-readable description. Ultimately, this book is a testament to Emily's passion for sharing knowledge and empowering others. By the time you reach the final page, you will have gained a solid foundation in the world of large language
Page
6
models, armed with the confidence to embark on your own exciting experiments and projects. So, without further ado, let us embark on this captivating journey through the world of large language models. Join Emily as she illuminates the way, inspiring you to embrace the power of these remarkable creations and harness their potential to reshape the future. Thank you, my friend, for the time, effort and - ultimately - love you put into this. Andrea Olgiati Chief Engineer, Amazon SageMaker Santa Clara, May 2023 Contributors About the author Emily Webber is a principal machine learning (ML) specialist solutions architect at Amazon Web Services (AWS). In her more than five years at AWS, she has assisted hundreds of customers on their journey to ML in the cloud, specializing in distributed training for large language and vision models. She mentors ML solution architects, authors countless feature designs for SageMaker and AWS, and guides the Amazon SageMaker product and engineering teams on best practices in regard to ML and customers. Emily is widely known in the AWS community for a 16-video YouTube series
Page
7
(https://www.youtube.com/playlistlist=PLhr1KZpdzukcOr_6j_zmSrvYnLU tgqsZz) featuring SageMaker with 211,000 views, and for giving a keynote speech at O’Reilly AI London 2019 on a novel reinforcement learning approach she developed for public policy. Acknowledgment It seems almost impossible to thank all of the talented, passionate, hardworking, and kind people who have helped me in my journey to this very moment. And yet I would be remiss if I didn't try! To the Packt team, including Dhruv Kataria, Priyanka Soam, Aparna Ravikumar Nair, and Hemangi Lotlikar, thank you so much for all of your enthusiasm, careful checks, and for believing in me from the start. Ten years ago I would have never thought I'd be writing a book on artificial intelligence, and it truly took the whole village of us to pull it off. There are quite literally hundreds of Amazonians I would like to thank for supporting me and the content of this book. This is due to services they've scoped and developed, customers they've onboarded and businesses they've built, design patterns they've developed, techniques they've perfected, the content they've built, and so much more. In particular, I'd like to call out Nav Bhasin, Mani Khanuja, Mark Roy, Shelbee Eigenbrode, Dhawal Patel, Kanwaljit Khurmi, Farooq Sabir, Sean Morgan, and all of the rest of the wonderful ML SA's I love working with every day. There are so many other teams at Amazon whose passion has helped create this work: from engineering to product, business development to marketing, all of my
Page
8
friends in the field, documentation, start-ups, public sector, and across the world, what an honor it is to make this dream come true with you. To my customers, you truly bring everything to life! Whether we shared an immersion day, a workshop, a call, a talk, a blog post, or a Slack channel, it is such an honor to serve as your technology partner in so many. We truly do walk up every day thinking about how to optimize for your outcomes! I spent quite a few days working on this book during the Winter Applications of Computer Vision (WACV) Conference in Hawaii, in January 2023. This was during a workshop I'd hosted https://sites.google.com/view/wacv2023-workshop with Fouad Bousetouane, Corey Barrett, Mani Khanuja, Larry Davis, and more. To all of my friends at WACV, it was such a perfect place to hash out ideas and misconceptions with you, thanks for all of your support! Finally, I'd love to thank my loving husband and the rest of my family for all of their support throughout the winding road that's been my life. Who would have guessed that some crazy girl from Pennsylvania would be writing books on AI from Amazon? Not me! To my readers, you make this all possible. I hope you'll consider reaching out to me and connecting! I'm live on Twitch every Monday, so you can always hop on for a question or a chat. For the future, stay tuned. I have a lot more words in me yet to write. About the reviewer Falk Pollok is a senior software engineer in IBM Research Europe and a senior RSE for the MIT-IBM Watson AI Lab, specializing in foundation
Page
9
models and multimodal question answering. Falk was a member of the MIT-Harvard-Stanford team on the Defense Advanced Research Projects Agency (DARPA) Machine Common Sense (MCS) project, contributed to IBM Watson Core, Orchestrate, and ML, was the lead developer for IBM Sapphire, and founded IBM’s Engineering Excellence program. He holds a master’s degree in computer science from RWTH Aachen, leadership certificates from Cornell, and IBM’s highest developer profession rank. Moreover, he has published eight papers in top conferences such as NeurIPS, AAAI, and Middleware, has two patents, was named a Face of IBM Research, and received multiple awards, including IBM’s OTA and InfoWorld’s Open Source Software Awards (BOSSIE) award.
Page
10
Table of Contents Preface
Page
11
Part 1: Before Pretraining 1 An Introduction to Pretraining Foundation Models The art of pretraining and fine-tuning The Transformer model architecture and self- attention State-of-the-art vision and language models Top vision models as of April 2023 Contrastive pretraining and natural language supervision Top language models as of April 2023 Language technique spotlight – causal modeling and the scaling laws Encoders and decoders Summary References 2
Page
12
Dataset Preparation: Part One Finding a dataset and use case for foundation modeling Top pretraining use cases by industry Delta – how different is your dataset? Use the scaling laws to size your datasets Fundamentals – scaling laws of neural language models Bias detection and mitigation Enhancing your dataset – multil ingual, multimodal, and augmentations Summary References 3 Model Preparation Finding your best base model Starting with the smallest base model you can Trade-off – simplicity versus complexity Finding your pretraining loss function Pretraining loss functions in vision – ViT and CoCa
Page
13
Pretraining loss functions in language – Alexa Teacher Model Changing your pretraining loss function Solving for your model size Practical approaches to solving for your model size Not all scaling laws are created equal Planning future experiments Summary References
Page
14
Part 2: Configure Your Environment 4 Containers and Accelerators on the Cloud What are accelerators and why do they matter? Getting ready to use your accelerators How to use accelerators on AWS – Amazon SageMaker Optimizing accelerator performance Hyperparameters Infrastructure optimizations for accelerators on AWS Troubleshooting accelerator performance Summary References 5 Distr ibution Fundamentals Understanding key concepts – data and model parallelism What data parallel is all about
Page
15
What model parallel is all about Combining model and data parallel Distributed training on Amazon SageMaker Distributed training software SM DDP SMP library Advanced techniques to reduce GPU memory Tensor parallelism Optimizer state sharding Activation checkpointing Sharded data parallelism Bringing it all home with examples from models today Stable Diffusion – data parallelism at scale GPT-3 – model and data parallelism at scale Summary References 6
Page
16
Dataset Preparation: Part Two, the Data Loader Introducing the data loader in Python Building and testing your own data loader – a case study from Stable Diffusion Creating embeddings – tokenizers and other key steps for smart features Optimizing your data pipeline on Amazon SageMaker Transforming deep learning datasets at scale on AWS Summary References
Page
17
Part 3: Train Your Model 7 Finding the Right Hyperparameters Hyperparameters – batch size, learning rate, and more Key hyperparameters in vision and language Tuning strategies Hyperparameter tuning for foundation models Scaling up as a function of world size with SageMaker Tuning on a sample of your data and updating based on world size Summary References 8 Large-Scale Training on SageMaker Optimizing your script for SageMaker training Importing packages Argument parsing
Page
18
Top usability features for SageMaker training Warm pools for rapid experimentation SSM and SSH into training instances Track jobs and experiments to replicate results Summary References 9 Advanced Training Concepts Evaluating and improving throughput Calculating model TFLOPS Using Flash Attention to speed up your training runs Speeding up your jobs with compilation Integrating compilation into your PyTorch scripts Amazon SageMaker Training Compiler and Neo Best practices for compilation Running compiled models on Amazon’s Trainium and Inferentia custom hardware Solving for an optimal training time Summary References
Page
19
Part 4: Evaluate Your Model 10 Fine-Tuning and Evaluating Fine-tuning for language, text, and everything in between Fine-tuning a language-only model Fine-tuning vision-only models Fine-tuning vision-language models Evaluating foundation models Model evaluation metrics for vision Model evaluation metrics in language Model evaluation metrics in joint vision- language tasks Incorporating the human perspective with labeling through SageMaker Ground Truth Reinforcement learning from human feedback Summary References 11
Page
20
Detecting, Mit igating, and Monitoring Bias Detecting bias in ML models Detecting bias in large vision and language models Mitigating bias in vision and language models Bias mitigation in language – counterfactual data augmentation and fair loss functions Bias mitigation in vision – reducing correlation dependencies and solving sampling issues Monitoring bias in ML models Detecting, mitigating, and monitoring bias with SageMaker Clarify Summary References 12 How to Deploy Your Model What is model deployment? What is the best way to host my model? Model deployment options on AWS with SageMaker Why should I shrink my model, and how? Model compilation
The above is a preview of the first 20 pages. Register to read the complete e-book.
AI Reading Assistant
Whole-book reading guide from stratified index samples; jump to passages in the text
AI guide
【One-Line Pitch】
A practical field guide to pretraining and fine-tuning foundation models—vision, language, and multimodal—on AWS and SageMaker, from project ideation through data prep, distributed training, evaluation, and deployment. Best for ML engineers and technical leads who already know deep learning and now need to run large-scale training in the cloud.
【Book Arc】
- **Opening (~0%–10%)**: Frames why foundation models matter and asks the decision questions up front—when to pretrain versus fine-tune, how to value a project, and why the book anchors on conceptual fundamentals plus AWS/SageMaker practice.
- **Early (~10%–30%)**: Turns ideation into a plan: choosing use cases and datasets, judging how "different" your data is (delta), applying scaling laws to size data/model/compute, and detecting and mitigating bias.
- **Early–Middle (~30%–45%)**: Moves into infrastructure—accelerators on AWS, containers and instance choices, and the distributed-training toolbox (data, pipeline, and tensor parallelism, optimizer state sharding, communication libraries).
- **Middle (~45%–60%)**: Gets hands-on with data pipelines: iterators and data loaders (with a Stable Diffusion case study), batching, embeddings, and feeding training loops efficiently.
- **Late (~60%–85%)**: Covers training and adaptation—fine-tuning, evaluation, bias remediation and monitoring, deployment options (real-time, serverless, batch, distributed hosting), and prompt engineering including prompt/prefix tuning.
- **Ending (~85%–100%)**: Closes with MLOps for vision and language—operations, orchestration, and pipeline monitoring—so models stay healthy in production.
【Key Takeaways】
- **Pretraining is a business decision before a technical one** (Early): the book pushes you to quantify whether an accuracy gain is worth hundreds of thousands or millions of dollars before committing to training from scratch.
- **Scaling laws guide sizing** (Early): use them to reason about data, model, and compute budgets together—most teams treat compute as fixed and work backward—rather than guessing.
- **Know how unique your dataset is** (Early): a "delta" analysis tells you which existing models are viable starting points and what performance ceiling to expect.
- **Bias is a lifecycle concern** (Early–Late): detect and mitigate it in data, then keep monitoring after deployment to reduce harm in real applications.
- **Distributed training is a menu of parallelism strategies** (Middle): data, pipeline, and tensor parallelism plus optimizer state sharding each fit different model sizes; keep a layer on one node to preserve bandwidth.
- **Managed libraries buy real efficiency** (Middle): SageMaker distributed training libraries and AWS deep learning containers encode hard-won optimization—skipping them leaves gains on the table.
- **Data loaders are the practical bottleneck** (Middle): understand iterators and framework loaders (PyTorch, TensorFlow, Hugging Face, NumPy) so batches actually feed the training loop.
- **Deployment starts with shrinking the model** (Late): try compression first; fall back to distributed hosting only when you can't or needn't shrink, then optimize serving end to end.
【Reading Tips】
- Deep-read the early chapters on use-case selection, scaling laws, and dataset delta—they determine whether the rest of the project is worth doing.
- Skim the accelerator/infrastructure chapter if you're not on AWS; keep the parallelism concepts, skip service-specific setup.
- Treat the data-loader and distributed-training chapters as reference while coding; the syntax will age, the concepts won't.
- Read the bias, evaluation, and MLOps chapters together as one production-readiness thread.
- Take away one artifact: a sizing and go/no-go memo for your own project.
【Coverage Limits】
Excerpts are uneven—strong on framing, sizing, and distributed training, thinner on fine-tuning, evaluation metrics, and MLOps specifics. Chapter-level detail beyond what's summarized here is not covered.
Passage locations
Excerpt 1
, achieve high throughput, and build deployment pipelines. By the end of this book, you'll be well equipped to embark on your own project to pretrain and fin...
View in text
Excerpt 2
y of “Bias” in NLP: https://arxiv.org/pdf/2005.14050.pdf 13. Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361.pdf 14. PaLM: Scaling...
View in text
Excerpt 3
ms will consider the compute budget as fixed. Think of this number as what you should plan on asking your senior leadership to approve for your project. As w...
View in text
Excerpt 4
For the rest of this book, however, we will mostly focus on dense networks that are applicable for models up to a few hundred billion parameters. Optimizer s...
View in text
Recommended for You
{{#thumbnailUrl}}
{{/thumbnailUrl}}
{{^thumbnailUrl}}
{{/thumbnailUrl}}
Loading recommended books...
Failed to load, please try again later
Tip the Site
Scan the WeChat Pay or Alipay code to tip. No login required.
WeChat Pay
Alipay