Pro Python System Administration - Learn to manage and monitor your network, web servers, and databases with Python (Rytis Sileika [Sileika R.]) (z-library.sk, 1lib.sk, z-lib.sk)

Sileika Python System Adm inistration Companion eBook Available 7.5 x 9.25 spine = 0.78125" 416 page count THE EXPERT’S VOICE® IN OPEN SOURCE Pro Python System Administration Rytis Sileika Learn to manage and monitor your network, web servers, and databases with Python this print for content only—size & color not accurate CYAN MAGENTA YELLOW BLACK PANTONE 123 C BOOKS FOR PROFESSIONALS BY PROFESSIONALS® Pro Python System Administration Dear Reader, Pro Python System Administration takes you beyond standard management tools and quick-and-dirty shell scripts. You’ll discover real-world examples of customized tools built using Python and the Django and Jinja2 frameworks – tools that you can put to work in your own environment right away. Each chapter explains how to determine your requirements, how to create your tools, and how to use them effectively. I chose the Python programming language because it's relatively simple to use, easy to learn, and elegant. It also comes with a large base of additional modules. These are some of the practical solutions you’ll learn to create: • How to manage network devices with SOAP and SNMP • How to monitor web servers and databases using Django and Jinja2 • How to perform statistical computation with NumPy • How to offload processing tasks to Amazon Web Services Administering network systems, a complex task in itself, requires a variety of specialized skills. This book will help you build the customized solutions you need to create an efficient, productive environment. Rytis Sileika, RHCE Rytis Sileika Shelve in: Python User level: Intermediate–Advanced THE APRESS ROADMAP Pro Python Pro Python System Administration Beginning Python Visualization Foundations of Agile Python Development Python 3 for Absolute Beginners Dive into Python 3 Beginning Python, Second Edition www.apress.com SOURCE CODE ONLINE Companion eBook See last page for details on $10 eBook version ISBN 978-1-4302-2605-5 9 781430 226055 54999 Pro

Pro Python System Administration ■ ■ ■ Rytis Sileika

Pro Python System Administration Copyright © 2010 by Rytis Sileika All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (pbk): 978-1-4302-2605-5 ISBN-13 (electronic): 978-1-4302-2606-2 Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1 Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. President and Publisher: Paul Manning Lead Editors: Duncan Parkes and Michelle Lowman Technical Reviewer: Patrick Engebretson Editorial Board: Clay Andres, Steve Anglin, Mark Beckner, Ewan Buckingham, Gary Cornell, Jonathan Gennick, Jonathan Hassell, Michelle Lowman, Matthew Moodie, Duncan Parkes, Jeffrey Pepper, Frank Pohlmann, Douglas Pundick, Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh Coordinating Editors: Mary Tobin and Jennifer L. Blackwell Copy Editors: Jim Compton, Heather Lang and Marilyn Smith Compositor: Lynn L’Heureux Indexer: Julie Grady Artist: April Milne Cover Designer: Anna Ishchenko Distributed to the book trade worldwide by Springer Science+Business Media, LLC., 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. For information on translations, please e-mail rights@apress.com, or visit www.apress.com. Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at www.apress.com/info/bulksales. The information in this book is distributed on an “as is” basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work. The source code for this book is available to readers at www.apress.com.

I want to dedicate this book to my family—my wife Evelina and daughters Gabija and Milda

■CONTENTS v Contents at a Glance ■About the Author................................................................................................................. xvi ■About the Technical Reviewer........................................................................................... xvii ■Acknowledgements .......................................................................................................... xviii ■Introduction.........................................................................................................................xix ■Chapter 1: Reading and Collecting Performance Data Using SNMP....................................1 ■Chapter 2: Managing Devices Using the SOAP API.............................................................41 ■Chapter 3: Creating a Web Application for IP Address Accountancy ................................75 ■Chapter 4: Integrating the IP Address Application with DHCP.........................................107 ■Chapter 5: Maintaining a List of Virtual Hosts in an Apache Configuration File .............137 ■Chapter 6: Gathering and Presenting Statistical Data from Apache Log Files ....................159 ■Chapter 7: Performing Complex Searches and Reporting on Application Log Files .......181 ■Chapter 8: A Web Site Availability Check Script for Nagios.............................................211 ■Chapter 9: Management and Monitoring Subsystem .......................................................231 ■Chapter 10: Remote Monitoring Agents............................................................................269 ■Chapter 11: Statistics Gathering and Reporting...............................................................297 ■Chapter 12: Automatic MySQL Database Performance Tuning........................................329 ■Chapter 13: Using Amazon EC2/S3 as a Data Warehouse Solution .................................349 ■Index...................................................................................................................................373

■CONTENTS vii Contents ■About the Author................................................................................................................. xvi ■About the Technical Reviewer........................................................................................... xvii ■Acknowledgements .......................................................................................................... xviii ■Introduction.........................................................................................................................xix ■Chapter 1: Reading and Collecting Performance Data Using SNMP....................................1 Application Requirements and Design ......................................................................................1 Specifying the Requirements .............................................................................................1 High-Level Design Specification.........................................................................................2 Introduction to SNMP ...............................................................................................................2 The System SNMP Variables Node .....................................................................................4 The Interfaces SNMP Variables Node .................................................................................5 Authentication in SNMP .....................................................................................................7 Querying SNMP from the Command Line ...........................................................................7 Querying SNMP Devices from Python .....................................................................................11 Configuring the Application ..............................................................................................11 Using the PySNMP Library ...............................................................................................13 Implementing the SNMP Read Functionality .....................................................................18 Storing Data with RRDTool .....................................................................................................18 Introduction to RRDTool ...................................................................................................18 Using RRDTool from a Python Program ............................................................................20 Creating a RoundRobin Database.....................................................................................20 Writing and Reading Data from the RoundRobin Database ...............................................22 Plotting Graphs with RRDTool ..........................................................................................25 Integrating RRDTool with the Monitoring Solution.............................................................28 Creating Web Pages with the Jinja2 Templating System.........................................................31 Loading Template Files with Jinja2 ..................................................................................32 The Jinja2 Template Language ........................................................................................33 Generating Web Site Pages..............................................................................................35 Summary ...............................................................................................................................39

■CONTENTS viii ■Chapter 2: Managing Devices Using the SOAP API.............................................................41 What Is the SOAP API?............................................................................................................41 The Structure of a SOAP Message....................................................................................42 Requesting Services with SOAP .......................................................................................42 Finding Information about Available Services with WSDL..................................................44 SOAP Support in Python .........................................................................................................45 Converting WSDL Schema to Python Helper Module ...............................................................46 Defining Requirements for Our Load Balancer Tool .................................................................47 Basic Requirements.........................................................................................................48 Code Structure.................................................................................................................48 Configuration ...................................................................................................................49 Accessing Citrix Netscaler Load Balancer with the SOAP API ..................................................50 Fixing Issues with Citrix Netscaler WSDL..........................................................................50 Creating a Connection Object...........................................................................................51 Logging In: Our First SOAP Call ........................................................................................54 Gathering Performance Statistics Data ...................................................................................59 SOAP Methods for Reading Statistical Data and Their Return Values ................................59 Reading System Health Data............................................................................................60 Reading Service Status Data ............................................................................................62 Automating Administration Tasks ...........................................................................................65 Device Configuration SOAP Methods ................................................................................66 Setting a Service State.....................................................................................................66 A Word About Logging and Error Handling ..............................................................................68 Using the Python logging Module..................................................................................68 Handling Exceptions ........................................................................................................72 Summary ...............................................................................................................................73 ■Chapter 3: Creating a Web Application for IP Address Accountancy ................................75 Designing the Application .......................................................................................................75 Setting Out the Requirements ..........................................................................................75 Making Design Decisions.................................................................................................76 Defining the Database Schema ........................................................................................77 Creating the Application Workflow ...................................................................................79

■CONTENTS ix The Basic Concepts of the Django Framework........................................................................80 What Is Django?...............................................................................................................81 The Model/View/Controller Pattern ...................................................................................81 Installing the Django Framework......................................................................................82 The Structure of a Django Application ..............................................................................83 Using Django with Apache Web Server.............................................................................85 Implementing Basic Functionality ...........................................................................................88 Defining the Database Model ...........................................................................................88 URL Configuration............................................................................................................91 Using the Management Interface .....................................................................................93 Viewing Records ..............................................................................................................96 Using Templates ..............................................................................................................99 Deleting Records ...........................................................................................................102 Adding New Records......................................................................................................103 Modifying Existing Records ............................................................................................105 Summary .............................................................................................................................106 ■Chapter 4: Integrating the IP Address Application with DHCP.........................................107 Extending the Design and Requirements ..............................................................................107 Extending the Database Schema....................................................................................109 Additions to the Workflow ..............................................................................................110 Adding DHCP Network Data..................................................................................................110 Defining Data Models.....................................................................................................110 Additional Workflows .....................................................................................................111 The Add Function...........................................................................................................111 The Modify Function ......................................................................................................113 The Delete Function.......................................................................................................113 Extending DHCP Configuration with Address Pools ...............................................................113 The Address Pool Data Model ........................................................................................114 Displaying DHCP Network Details...................................................................................114 The Add and Delete Functions........................................................................................116 Reworking the URL Structure ...............................................................................................116 Generating URLs in the Model Class...............................................................................117 Reverse Resolution of URLs ...........................................................................................117 Assigning Names to URL Patterns ..................................................................................118 Using URL References in the Templates.........................................................................118

■CONTENTS x Adding Client Classification ..................................................................................................120 Additions to the Data Model . ..........................................................................................120 Using Template Inheritance. ...........................................................................................121 Class Rules Management. ..............................................................................................122 Generating the DHCP Configuration File................................................................................127 Other Modifications ..............................................................................................................131 Resolving IPs to Hostnames . ..........................................................................................131 Checking Whether the Address Is In Use . .......................................................................132 Summary .............................................................................................................................135 ■Chapter 5: Maintaining a List of Virtual Hosts in an Apache Configuration File .............137 Specifying the Design and Requirements for the Application.................................................137 Functional Requirements . ..............................................................................................138 High-Level Design . ........................................................................................................138 Setting Up the Environment ..................................................................................................138 Apache Configuration. ....................................................................................................139 Creating a Django Project and Application. .....................................................................139 Configuring the Application . ...........................................................................................140 Defining the URL Structure. ............................................................................................141 The Data Model....................................................................................................................142 The Basic Model Structure . ............................................................................................143 Modifying the Administration Interface . ................................................................................147 Improving the Class and Object Lists . ............................................................................147 Adding Custom Object Actions .......................................................................................152 Generating the Configuration File..........................................................................................154 Summary .............................................................................................................................157 ■Chapter 6: Gathering and Presenting Statistical Data from Apache Log Files ....................159 Application Structure and Functionality.................................................................................159 Application Requirements . .............................................................................................160 Application Design . ........................................................................................................160 Plug-in Framework Implementation in Python ......................................................................160 The Mechanics of a Plug-in Framework . ........................................................................161 Creating the Plug-in Framework . ...................................................................................163

■CONTENTS xi Log-Parsing Application .......................................................................................................167 Format of Apache Log Files............................................................................................167 Log File Reader..............................................................................................................169 Calling the Plug-in Methods ...........................................................................................173 Plug-in Modules ...................................................................................................................177 Installing the Required Libraries.....................................................................................177 Writing the Plug-in Code ................................................................................................179 Summary .............................................................................................................................180 ■Chapter 7: Performing Complex Searches and Reporting on Application Log Files .......181 Defining the Problem............................................................................................................181 Why We Use Exceptions.................................................................................................184 Are Exceptions Always a Bad Sign?................................................................................184 Why We Should Analyze Exceptions ...............................................................................184 Parsing Complex Log Files ...................................................................................................186 What Can We Find in a Typical Log File? ........................................................................186 The Structure of an Exception Stack Trace Log ..............................................................187 Handling Multiple Files .........................................................................................................189 Handling Multiple Files...................................................................................................189 Using the Built In Bzip2 Library ......................................................................................193 Traversing Through Large Data Files..............................................................................193 What Are Generators, And How Do We Use Them?.........................................................193 Detecting Exceptions............................................................................................................195 Detecting Potential Candidates ......................................................................................195 Filtering Legitimate Exception Traces.............................................................................196 Storing Data in Data Structures ............................................................................................197 The Structure of Exception Stack Trace Data .................................................................198 Generating an Exception Fingerprint for Unknown Exceptions ........................................198 Detecting Known Exceptions..........................................................................................201 Producing Reports................................................................................................................206 Grouping Exceptions ......................................................................................................207 Producing Differently Formatted Output for the Same Dataset........................................208 Calculating Group Statistics ...........................................................................................208 Summary .............................................................................................................................210

■CONTENTS xii ■Chapter 8: A Web Site Availability Check Script for Nagios.............................................211 Requirements for the Check System.....................................................................................211 The Nagios Monitoring System .............................................................................................212 Nagios Plug-In Architecture ...........................................................................................212 The Site Navigation Check....................................................................................................213 Installing the Beautiful Soup HTML Parsing Library ........................................................214 Retrieving a Web Page...................................................................................................214 Parsing the HTML Pages with Beautiful Soup .................................................................216 Adding the New Check to the Nagios System.................................................................222 Emulating the User Login Process..................................................................................224 Summary .............................................................................................................................229 ■Chapter 9: Management and Monitoring Subsystem .......................................................231 Design .................................................................................................................................231 The Components............................................................................................................231 The Data Objects ...........................................................................................................233 The Data Structures .............................................................................................................234 Introduction to Data Normalization .................................................................................235 Configuration Data .........................................................................................................237 Performance Data..........................................................................................................240 Scheduling ....................................................................................................................241 Site Configuration ..........................................................................................................242 Communication Flows ..........................................................................................................243 XML-RPC for Information Exchange ...............................................................................244 CherryPy........................................................................................................................246 The Server Process ..............................................................................................................247 Storing Data in a SQLite3 Database ...............................................................................247 Actions ..........................................................................................................................252 The Scheduler ......................................................................................................................255 Actions ..........................................................................................................................255 Running Multiple Processes ...........................................................................................255 Running Methods at Equal Intervals ...............................................................................260 A Cron-Like Scheduler ...................................................................................................263 Ticket Dispatcher...........................................................................................................264 Summary .............................................................................................................................267

■CONTENTS xiii ■Chapter 10: Remote Monitoring Agents............................................................................269 Design .................................................................................................................................269 The Passive Component.................................................................................................269 Architecture ...................................................................................................................269 Actions ..........................................................................................................................269 The Security Model ..............................................................................................................271 Configuration........................................................................................................................271 The ConfigParser Library................................................................................................271 The Configuration Class Wrapper ...................................................................................277 The Sensor Design ...............................................................................................................281 Running External Processes .................................................................................................282 Using the subprocess Library .........................................................................................282 Controlling the Running Processes.................................................................................286 Communicating with External Processes........................................................................289 Automatically Updating Sensor Code ....................................................................................292 Sending and Receiving Binary Data with XML-RPC.........................................................292 Working with Files and Archives (TAR and BZip2)...........................................................293 Summary .............................................................................................................................296 ■Chapter 11: Statistics Gathering and Reporting...............................................................297 Application Requirements and Design ..................................................................................297 Using the NumPy Library ......................................................................................................297 Installing NumPy............................................................................................................298 NumPy Examples...........................................................................................................298 Representing Data with matplotlib ........................................................................................309 Installing matplotlib .......................................................................................................309 Understanding the Library Structure...............................................................................310 Plotting Graphs ..............................................................................................................311 Saving Plots to a File......................................................................................................316 Graphing Statistical Data ......................................................................................................317 Collating Data from the Database...................................................................................317 Drawing Timescale Graphs ............................................................................................318 Summary .............................................................................................................................328

■CONTENTS xiv ■Chapter 12: Automatic MySQL Database Performance Tuning........................................329 Requirements Specification and Design................................................................................329 Basic Application Requirements.....................................................................................330 System Design...............................................................................................................330 Modifying the Plug-in Framework.........................................................................................332 Changes to the Host Application.....................................................................................332 Modifying the Plug-in Manager ......................................................................................333 Writing the Producer Plug-ins...............................................................................................334 Accessing the MySQL Database from Python Applications..............................................334 Querying the Configuration Variables .............................................................................339 Querying the Server Status Variables .............................................................................341 Collecting the Host Configuration Data ...........................................................................342 Writing the Consumer Plug-ins .............................................................................................343 Checking the MySQL Version .........................................................................................344 Checking the Key Buffer Size Setting .............................................................................346 Checking the Slow Queries Counter ...............................................................................347 Summary .............................................................................................................................348 ■Chapter 13: Using Amazon EC2/S3 as a Data Warehouse Solution .................................349 Specifying the Problem and the Solution...............................................................................349 The Problem ..................................................................................................................349 Our Solution...................................................................................................................350 Design Specifications.....................................................................................................350 The Amazon EC2 and S3 Crash Course.................................................................................351 Authentication and Security ...........................................................................................351 The Simple Storage System Concepts............................................................................352 The Elastic Computing Cloud Concepts ..........................................................................354 User Interfaces ..............................................................................................................358 Creating a Custom EC2 Image ..............................................................................................359 Reusing Existing Images ................................................................................................359 Making Modifications.....................................................................................................360 Bundling the New AMI ...................................................................................................362

■CONTENTS xv Controlling the EC2 Using the Boto Python Module ...............................................................364 Setting Up the Configuration Variables ...........................................................................364 Initializing the EC2 Instance Programmatically ...............................................................365 Transferring the Data .....................................................................................................369 Destroying the EC2 Instance Programmatically ..............................................................369 Summary .............................................................................................................................372 ■Index...................................................................................................................................373

■CONTENTS xvi About the Author ■Ry tis S ileika has over twelve years of experience in the system administration field. Since obtaining his bachelor of science degree in computer science from Kaunas University of Technology, he’s been specializing in system integration and deployment automation. His areas of interest and expertise are UNIX-based operating system management and automation tool development. Rytis is also a RedHat Certified Engineer. He lives with his wife and two daughters in London, United Kingdom. His nonprofessional interests are traveling, hiking, and photography.

■CONTENTS xvii About the Technical Reviewer ■Dr. P atr ic k Engebres ton obtained his doctor of science degree with a specialization in information assurance from Dakota State University. He currently serves as an assistant professor of computer and network security and works as a senior penetration tester for security firm in the Midwest. His research interests include penetration testing, intrusion detection, exploitation, malware, and programming. He teaches courses in security, C programming, and Python. When not hacking or teaching, Dr. Engebretson spends every waking minute with his wife Lori and his two beautiful girls Maggie and Molly.

■CONTENTS xviii Acknowledgments I’d like to express my gratitude to everyone at Apress involved in the development and production of this book. First, I want to thank Duncan Parkes, who helped a lot with the initial proposal, set the general shape and structure of the book, and got the whole project moving forward. Many thanks go to Michelle Lowman and Dr. Patrick Engebretson for correcting all technical and logical mistakes as well as providing valuable tips. I would also like to thank Jennifer Blackwell and Mary Tobin for keeping the project and my writing on schedule and gently reminding me about the approaching deadlines. Last but not least, I’d like to thank the Python development community and Guido van Rossum for creating such a nice and elegant programming language.