Name: Math for Programmers (Paul Orland) (z-library.sk, 1lib.sk, z-lib.sk)
Author: Unknown Author

Page 1

M A N N I N G Paul Orland 3D graphics, machine learning, and simulations with Python

Page 2

3Mathematical Notation Reference Mathematical Notation Reference (continues on inside back cover) Notation example Name Defined in section (x, y) 2.1.1 sin(x) 2.3.1 cos(x) 2.3.1 θ 2.3.1 π 2.3.2 (x, y, z) 3.1.1 u · v 3.3 u× v 3.4 f(g(x)) 4.1.2 f(x, y) versus f(x)(y) 4.1.2 au+ bv 4.2.3 e1, e2, e3, . . . 4.2.4 v = ⎛ ⎝1 2 3 ⎞ ⎠ 5.1.1 A = ⎛ ⎝1 2 3 4 5 6 7 8 9 ⎞ ⎠ 5.1.1 R n 6.2.1 (f + g)(x) or f(x) + g(x) 6.2.3 c · f(x) 6.2.3 span({u,v,w}) 6.3.3 f(x) = ax+ b 6.3.5 f(x) = a0 + a1x+ · · ·+ anx n Coordinate vector in 2D Trigonometric sine function Trigonometric cosine function Greek letter theta, commonly representing angle measure Greek letter pi, representing the number 3.14159 . . . Coordinate vector in 3D Dot product of two vectors u and v Cross product of two vectors u and v Composition of functions See discussion of currying. Linear combination two vectors u and v Standard basis vectors Column vector Matrix Real coordinate vector space of dimension n Adding two functions Scalar multiplication of a function The span of a set of vectors Linear function Polynomial function 6.3.5

Page 3

Math for Programmers 3D graphics, machine learning and simulations with Python

Page 4

ii

Page 5

Math for Programmers 3D GRAPHICS, MACHINE LEARNING AND SIMULATIONS WITH PYTHON PAUL ORLAND M A N N I N G SHELTER ISLAND

Page 6

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2020 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Jenny Stout 20 Baldwin Road Technical development editor: Kris Athi PO Box 761 Review editor: Aleks Dragosavljević Shelter Island, NY 11964 Production editor: Lori Weidert Copy editor: Frances Buran Proofreader: Jason Everett Technical proofreader: Mike Shepard Typesetter and cover designer: Marija Tudor ISBN 9781617295355 Printed in the United States of America

Page 7

To my first math teacher and my first programming teacher—Dad.

Page 8

vi

Page 9

brief contents 1 ■ Learning math with code 1 PART 1 VECTORS AND GRAPHICS ............................................... 19 2 ■ Drawing with 2D vectors 21 3 ■ Ascending to the 3D world 75 4 ■ Transforming vectors and graphics 121 5 ■ Computing transformations with matrices 158 6 ■ Generalizing to higher dimensions 205 7 ■ Solving systems of linear equations 257 PART 2 CALCULUS AND PHYSICAL SIMULATION ........................ 301 8 ■ Understanding rates of change 303 9 ■ Simulating moving objects 337 10 ■ Working with symbolic expressions 354 11 ■ Simulating force fields 392 12 ■ Optimizing a physical system 422 13 ■ Analyzing sound waves with a Fourier series 463 PART 3 MACHINE LEARNING APPLICATIONS ............................. 497 14 ■ Fitting functions to data 499 15 ■ Classifying data with logistic regression 526 16 ■ Training neural networks 559vii

Page 10

BRIEF CONTENTSviii

Page 11

contents preface xvii acknowledgments xxi about this book xxiii about the author xxviii about the cover illustration xxix 1 Learning math with code 1 1.1 Solving lucrative problems with math and software 2 Predicting financial market movements 2 ■ Finding a good deal 5 ■ Building 3D graphics and animations 7 Modeling the physical world 9 1.2 How not to learn math 11 Jane wants to learn some math 12 ■ Slogging through math textbooks 13 1.3 Using your well-trained left brain 13 Using a formal language 14 ■ Build your own calculator 15 Building abstractions with functions 16 PART 1 VECTORS AND GRAPHICS ................................ 19 2 Drawing with 2D vectors 21 2.1 Picturing 2D vectors 22 Representing 2D vectors 24 ■ 2D drawing in Python 26 Exercises 29ix

Page 12

CONTENTSx2.2 Plane vector arithmetic 32 Vector components and lengths 35 ■ Multiplying vectors by numbers 37 ■ Subtraction, displacement, and distance 39 Exercises 42 2.3 Angles and trigonometry in the plane 51 From angles to components 52 ■ Radians and trigonometry in Python 56 ■ From components back to angles 57 Exercises 60 2.4 Transforming collections of vectors 67 Combining vector transformations 69 ■ Exercises 70 2.5 Drawing with Matplotlib 72 3 Ascending to the 3D world 75 3.1 Picturing vectors in 3D space 77 Representing 3D vectors with coordinates 79 ■ 3D drawing with Python 80 ■ Exercises 82 3.2 Vector arithmetic in 3D 83 Adding 3D vectors 83 ■ Scalar multiplication in 3D 85 Subtracting 3D vectors 85 ■ Computing lengths and distances 86 Computing angles and directions 87 ■ Exercises 89 3.3 The dot product: Measuring vector alignment 92 Picturing the dot product 93 ■ Computing the dot product 95 Dot products by example 97 ■ Measuring angles with the dot product 97 ■ Exercises 100 3.4 The cross product: Measuring oriented area 103 Orienting ourselves in 3D 103 ■ Finding the direction of the cross product 106 ■ Finding the length of the cross product 108 Computing the cross product of 3D vectors 109 ■ Exercises 110 3.5 Rendering a 3D object in 2D 114 Defining a 3D object with vectors 114 ■ Projecting to 2D 116 Orienting faces and shading 116 ■ Exercises 119 4 Transforming vectors and graphics 121 4.1 Transforming 3D objects 123 Drawing a transformed object 124 ■ Composing vector transformations 126 ■ Rotating an object about an axis 129 Inventing your own geometric transformations 131 Exercises 134

Page 13

CONTENTS xi4.2 Linear transformations 138 Preserving vector arithmetic 138 ■ Picturing linear transformations 140 ■ Why linear transformations? 142 Computing linear transformations 146 ■ Exercises 149 5 Computing transformations with matrices 158 5.1 Representing linear transformations with matrices 159 Writing vectors and linear transformations as matrices 159 Multiplying a matrix with a vector 161 ■ Composing linear transformations by matrix multiplication 163 ■ Implementing matrix multiplication 166 ■ 3D animation with matrix transformations 166 ■ Exercises 169 5.2 Interpreting matrices of different shapes 175 Column vectors as matrices 176 ■ What pairs of matrices can be multiplied? 178 ■ Viewing square and non-square matrices as vector functions 180 ■ Projection as a linear map from 3D to 2D 181 ■ Composing linear maps 184 Exercises 186 5.3 Translating vectors with matrices 191 Making plane translations linear 191 ■ Finding a 3D matrix for a 2D translation 194 ■ Combining translation with other linear transformations 195 ■ Translating 3D objects in a 4D world 196 ■ Exercises 199 6 Generalizing to higher dimensions 205 6.1 Generalizing our definition of vectors 206 Creating a class for 2D coordinate vectors 207 ■ Improving the Vec2 class 208 ■ Repeating the process with 3D vectors 209 Building a vector base class 210 ■ Defining vector spaces 212 Unit testing vector space classes 214 ■ Exercises 216 6.2 Exploring different vector spaces 219 Enumerating all coordinate vector spaces 219 ■ Identifying vector spaces in the wild 221 ■ Treating functions as vectors 223 Treating matrices as vectors 226 ■ Manipulating images with vector operations 227 ■ Exercises 230 6.3 Looking for smaller vector spaces 237 Identifying subspaces 238 ■ Starting with a single vector 240 Spanning a bigger space 240 ■ Defining the word dimension 243 ■ Finding subspaces of the vector space of functions 244 ■ Subspaces of images 245 ■ Exercises 248

Page 14

CONTENTSxii7 Solving systems of linear equations 257 7.1 Designing an arcade game 258 Modeling the game 259 ■ Rendering the game 260 Shooting the laser 261 ■ Exercises 262 7.2 Finding intersection points of lines 263 Choosing the right formula for a line 263 ■ Finding the standard form equation for a line 265 ■ Linear equations in matrix notation 267 ■ Solving linear equations with NumPy 268 Deciding whether the laser hits an asteroid 270 ■ Identifying unsolvable systems 271 ■ Exercises 273 7.3 Generalizing linear equations to higher dimensions 278 Representing planes in 3D 278 ■ Solving linear equations in 3D 280 ■ Studying hyperplanes algebraically 282 ■ Counting dimensions, equations, and solutions 283 ■ Exercises 285 7.4 Changing basis by solving linear equations 294 Solving a 3D example 296 ■ Exercises 297 PART 2 CALCULUS AND PHYSICAL SIMULATION ......... 301 8 Understanding rates of change 303 8.1 Calculating average flow rate from volume 305 Implementing an average_flow_rate function 305 ■ Picturing the average flow rate with a secant line 306 ■ Negative rates of change 308 ■ Exercises 309 8.2 Plotting the average flow rate over time 310 Finding the average flow rate in different time intervals 310 Plotting the interval flow rates 311 ■ Exercises 313 8.3 Approximating instantaneous flow rates 315 Finding the slope of small secant lines 315 ■ Building the instantaneous flow rate function 318 ■ Currying and plotting the instantaneous flow rate function 320 ■ Exercises 322 8.4 Approximating the change in volume 323 Finding the change in volume for a short time interval 323 Breaking up time into smaller intervals 324 ■ Picturing the volume change on the flow rate graph 325 ■ Exercises 328 8.5 Plotting the volume over time 328 Finding the volume over time 328 ■ Picturing Riemann sums for the volume function 329 ■ Improving the approximation 332 Definite and indefinite integrals 334

Page 15

CONTENTS xiii9 Simulating moving objects 337 9.1 Simulating a constant velocity motion 338 Adding velocities to the asteroids 339 ■ Updating the game engine to move the asteroids 339 ■ Keeping the asteroids on the screen 340 ■ Exercises 342 9.2 Simulating acceleration 342 Accelerating the spaceship 343 9.3 Digging deeper into Euler’s method 344 Carrying out Euler’s method by hand 344 ■ Implementing the algorithm in Python 346 9.4 Running Euler’s method with smaller time steps 348 Exercises 349 10 Working with symbolic expressions 354 10.1 Finding an exact derivative with a computer algebra system 355 Doing symbolic algebra in Python 356 10.2 Modeling algebraic expressions 358 Breaking an expression into pieces 358 ■ Building an expression tree 359 ■ Translating the expression tree to Python 360 Exercises 362 10.3 Putting a symbolic expression to work 365 Finding all the variables in an expression 365 ■ Evaluating an expression 366 ■ Expanding an expression 369 ■ Exercises 372 10.4 Finding the derivative of a function 374 Derivatives of powers 374 ■ Derivatives of transformed functions 375 ■ Derivatives of some special functions 377 Derivatives of products and compositions 378 ■ Exercises 379 10.5 Taking derivatives automatically 381 Implementing a derivative method for expressions 382 Implementing the product rule and chain rule 383 Implementing the power rule 384 ■ Exercises 386 10.6 Integrating functions symbolically 387 Integrals as antiderivatives 387 ■ Introducing the SymPy library 388 ■ Exercises 389 11 Simulating force fields 392 11.1 Modeling gravity with a vector field 393 Modeling gravity with a potential energy function 394

Page 16

CONTENTSxiv11.2 Modeling gravitational fields 396 Defining a vector field 396 ■ Defining a simple force field 398 11.3 Adding gravity to the asteroid game 399 Making game objects feel gravity 400 ■ Exercises 403 11.4 Introducing potential energy 404 Defining a potential energy scalar field 405 ■ Plotting a scalar field as a heatmap 407 ■ Plotting a scalar field as a contour map 407 11.5 Connecting energy and forces with the gradient 408 Measuring steepness with cross sections 409 ■ Calculating partial derivatives 411 ■ Finding the steepness of a graph with the gradient 413 ■ Calculating force fields from potential energy with the gradient 415 ■ Exercises 418 12 Optimizing a physical system 422 12.1 Testing a projectile simulation 425 Building a simulation with Euler’s method 426 ■ Measuring properties of the trajectory 427 ■ Exploring different launch angles 428 ■ Exercises 429 12.2 Calculating the optimal range 432 Finding the projectile range as a function of the launch angle 432 Solving for the maximum range 435 ■ Identifying maxima and minima 437 ■ Exercises 439 12.3 Enhancing our simulation 440 Adding another dimension 441 ■ Modeling terrain around the cannon 442 ■ Solving for the range of the projectile in 3D 443 Exercises 447 12.4 Optimizing range using gradient ascent 449 Plotting range versus launch parameters 449 ■ The gradient of the range function 450 ■ Finding the uphill direction with the gradient 451 ■ Implementing gradient ascent 453 Exercises 457 13 Analyzing sound waves with a Fourier series 463 13.1 Combining sound waves and decomposing them 465 13.2 Playing sound waves in Python 466 Producing our first sound 467 ■ Playing a musical note 469 Exercises 471 13.3 Turning a sinusoidal wave into a sound 471

Page 17

CONTENTS xvMaking audio from sinusoidal functions 471 ■ Changing the frequency of a sinusoid 473 ■ Sampling and playing the sound wave 475 ■ Exercises 477 13.4 Combining sound waves to make new ones 478 Adding sampled sound waves to build a chord 478 ■ Picturing the sum of two sound waves 479 ■ Building a linear combination of sinusoids 481 ■ Building a familiar function with sinusoids 483 ■ Exercises 486 13.5 Decomposing a sound wave into its Fourier series 486 Finding vector components with an inner product 487 ■ Defining an inner product for periodic functions 488 ■ Writing a function to find Fourier coefficients 490 ■ Finding the Fourier coefficients for the square wave 491 ■ Fourier coefficients for other waveforms 492 ■ Exercises 494 PART 3 MACHINE LEARNING APPLICATIONS .............. 497 14 Fitting functions to data 499 14.1 Measuring the quality of fit for a function 502 Measuring distance from a function 503 ■ Summing the squares of the errors 505 ■ Calculating cost for car price functions 507 Exercises 510 14.2 Exploring spaces of functions 511 Picturing cost for lines through the origin 512 ■ The space of all linear functions 514 ■ Exercises 515 14.3 Finding the line of best fit using gradient descent 515 Rescaling the data 516 ■ Finding and plotting the line of best fit 516 ■ Exercises 518 14.4 Fitting a nonlinear function 519 Understanding the behavior of exponential functions 519 Finding the exponential function of best fit 521 ■ Exercises 523 15 Classifying data with logistic regression 526 15.1 Testing a classification function on real data 528 Loading the car data 529 ■ Testing the classification function 529 ■ Exercises 530 15.2 Picturing a decision boundary 532 Picturing the space of cars 532 ■ Drawing a better decision boundary 533 ■ Implementing the classification function 534 Exercises 535

Page 18

CONTENTSxvi15.3 Framing classification as a regression problem 536 Scaling the raw car data 536 ■ Measuring the “BMWness” of a car 538 ■ Introducing the sigmoid function 540 ■ Composing the sigmoid function with other functions 541 ■ Exercises 543 15.4 Exploring possible logistic functions 544 Parameterizing logistic functions 545 ■ Measuring the quality of fit for a logistic function 546 ■ Testing different logistic functions 548 ■ Exercises 549 15.5 Finding the best logistic function 551 Gradient descent in three dimensions 551 ■ Using gradient descent to find the best fit 552 ■ Testing and understanding the best logistic classifier 554 ■ Exercises 555 16 Training neural networks 559 16.1 Classifying data with neural networks 561 16.2 Classifying images of handwritten digits 562 Building the 64-dimensional image vectors 563 ■ Building a random digit classifier 565 ■ Measuring performance of the digit classifier 566 ■ Exercises 567 16.3 Designing a neural network 568 Organizing neurons and connections 568 ■ Data flow through a neural network 569 ■ Calculating activations 572 Calculating activations in matrix notation 574 ■ Exercises 576 16.4 Building a neural network in Python 577 Implementing an MLP class in Python 578 ■ Evaluating the MLP 580 ■ Testing the classification performance of an MLP 581 ■ Exercises 582 16.5 Training a neural network using gradient descent 582 Framing training as a minimization problem 582 ■ Calculating gradients with backpropagation 584 ■ Automatic training with scikit-learn 585 ■ Exercises 586 16.6 Calculating gradients with backpropagation 588 Finding the cost in terms of the last layer weights 589 Calculating the partial derivatives for the last layer weights using the chain rule 590 ■ Exercises 591 appendix A Getting set up with Python 595 appendix B Python tips and tricks 607 appendix C Loading and rendering 3D Models with OpenGL and PyGame 635 index 645

Page 19

preface I started working on this book in 2017, when I was CTO of Tachyus, a company I founded that builds predictive analytics software for oil and gas companies. By that time, we had finished building our core product: a fluid-flow simulator powered by physics and machine learning, along with an optimization engine. These tools let our customers look into the future of their oil reservoirs and helped them to discover hundreds of millions of dollars of optimization opportunities. My task as CTO was to productize and scale-out this software as some of the biggest companies in the world began to use it. The challenge was that this was not only a complex software project, but the code was very mathematical. Around that time, we started hiring for a position called “scientific software engineer,” with the idea that we needed skilled professional software engineers who also had solid backgrounds in math, physics, and machine learning. In the process of searching for and hiring scien- tific software engineers, I realized that this combination was both rare and in high demand. Our software engineers realized this as well and were eager to hone their math skills to contribute to our specialized back-end components of our stack. With eager math learners on our team already, as well as in our hiring pipeline, I started to think about the best way to train a strong software engineer to become a formidable math user. I realized there were no books with the right math content, presented at the right level. While there are probably hundreds of books and thousands of free online arti- cles on topics like linear algebra and calculus, I’m not aware of any I could hand to a typical professional software engineer, and expect them to come back in a few months having mastered the material. I don’t say this to disparage software engineers, I just mean that reading and understanding math books is a difficult skill to learn on its own. To do so, you often need to figure out what specific topics you need to learnxvii

Page 20

PREFACExviii(which is hard if you don’t know anything about the material yet!), read them, and then choose some high quality exercises to practice applying those topics. If you were less discerning, you could read every word of a textbook and solve all of its exercises, but it could take months of full-time study to do that! With Math for Programmers, I hope to offer an alternative. I believe it’s possible to read this book cover-to-cover in a reasonable amount of time, including completing all the exercises, and then to walk away having mastered some key math concepts. How this book was designed In the fall of 2017, I got in touch with Manning and learned that they were interested in publishing this book. That started a long process of converting my vision for this book into a concrete plan, which was much more difficult than I imagined, being a first-time author. Manning asked some hard questions of my original table of con- tents, like Will anyone be interested in this topic? Will this be too abstract? Can you really teach a semester of calculus in one chapter? All of these questions forced me to think a lot more carefully about what was achiev- able. I’ll share some of the ways we answered these questions because they’ll help you understand exactly how this book works. First, I decided to focus this book around one core skill—expressing mathematical ideas in code. I think this is a great way to learn math, even if you aren’t a programmer by trade. When I was in high school, I learned to program on my TI-84 graphing cal- culator. I had the grand idea that I could write programs to do my math and science homework for me, giving me the right answer and outputting the steps along the way. As you might expect, this was more difficult than just doing my homework in the first place, but it gave me some useful perspective. For any kind of problem I wanted to program, I had to clearly understand the inputs and outputs, and what happened in each of the steps of the solution. By the end, I was sure I knew the material, and I had a working program to prove it. That’s the experience I’ll try to share with you in this book. Each chapter is orga- nized around a tangible example program, and to get it working, you need to put all the mathematical pieces together correctly. Once you’re done, you’ll have confidence that you’ve understood the concept and can apply it again in the future. I’ve included plenty of exercises to help you check your understanding on the math and code I’ve included, as well as mini-projects which invite you to experiment with new variations on the material. Another question I discussed with Manning was what programming language I should use for the examples. Originally, I wanted to write the book in a functional programming language because math is a functional language itself. After all, the con- cept of a “function” originated in math, long before computers even existed. In vari- ous parts of math, you have functions that return other functions like integrals and

Math for Programmers (Paul Orland) (z-library.sk, 1lib.sk, z-lib.sk)

Math for Programmers (Paul Orland) (z-library.sk, 1lib.sk, z-lib.sk)

Registered users can read the full content for free

AI Reading Assistant

Passage locations

Support Author

Recommended for You