Network Flow Analysis (Michael W. Lucas)（Z-Library）

Page 1

(This page has no text content)

Page 2

(This page has no text content)

Page 3

PRAISE FOR ABSOLUTE FREEBSD, 2ND EDITION BY MICHAEL LUCAS “Michael Lucas is probably the best system administration author I’ve read. I am amazed that he can communicate top-notch content with a sense of humor, while not offending the reader or sounding stupid. When was the last time you could physically feel yourself getting smarter while reading a book?” —RICHARD BEJTLICH, TAOSECURITY “Master practitioner Lucas organizes features and functions to make sense in the development environment, and so provides aid and comfort to new users, novices, and those with significant experience alike.” —SCITECH BOOK NEWS PRAISE FOR ABSOLUTE OPENBSD BY MICHAEL LUCAS “A well-written book that hits its market squarely on target. Those new to OpenBSD will appreciate the comprehensive approach that takes them from concept to functional execution. Existing and advanced users will benefit from the discussion of OpenBSD-specific topics such as the security features and pf administration.” —SLASHDOT “The potentially boring topic of systems administration is made very readable and even fun by the light tone that Lucas uses.” —CHRIS PALMER, PRESIDENT, SAN FRANCISCO OPENBSD USERS GROUP “I recommend Absolute OpenBSD to all programmers and administrators working with the OpenBSD operating system (OS), or considering it.” —UNIXREVIEW.COM

Page 4

PRAISE FOR CISCO ROUTERS FOR THE DESPERATE, 2ND EDITION BY MICHAEL LUCAS “For me, reading this book was like having one of the guys in my company who lives and breathes Cisco sitting down with me for a day and explaining everything I need to know to handle problems or issues likely to come my way.” —IT WORLD “This really ought to be the book inside every Cisco Router box for the very slim chance things go goofy and help is needed ‘right now.’” —MACCOMPANION PRAISE FOR PGP & GPG BY MICHAEL LUCAS “Excellent tutorial, quick read, and enough humor to make it enjoyable.” —INFOWORLD “An excellent book that shows the end-user in an easy to read and often entertaining style just about everything they need to know to effectively and properly use PGP and OpenPGP.” —SLASHDOT “The world’s first user-friendly book on email privacy. Unless you’re a cryptographer, or never use email, you should read this book.” —LEN SASSAMAN, CODECON FOUNDER

Page 5

NETWORK FLOW ANALYSIS by Michael W. Lucas San Francisco

Page 6

NETWORK FLOW ANALYSIS. Copyright © 2010 by Michael W. Lucas. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. 14 13 12 11 10 1 2 3 4 5 6 7 8 9 ISBN-10: 1-59327-203-0 ISBN-13: 978-1-59327-203-6 Publisher: William Pollock Production Editor: Ansel Staton Cover and Interior Design: Octopod Studios Developmental Editor: William Pollock Technical Reviewer: Richard Bejtlich Copyeditor: Kim Wimpsett Compositors: Riley Hoffman and Ansel Staton Proofreader: Linda Seifert Indexer: Nancy Guenther For information on book distributors or translations, please contact No Starch Press, Inc. directly: No Starch Press, Inc. 38 Ringold Street, San Francisco, CA 94103 phone: 415.863.9900; fax: 415.863.9950; info@nostarch.com; www.nostarch.com Library of Congress Cataloging-in-Publication Data Lucas, Michael, 1967- Network flow analysis / by Michael W. Lucas. p. cm. Includes index. ISBN-13: 978-1-59327-203-6 ISBN-10: 1-59327-203-0 1. Network analysis (Planning)--Data processing. I. Title. T57.85.L83 2010 658.4'032--dc22 2010015790 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

Page 7

For Liz

Page 8

(This page has no text content)

Page 9

B R I E F C O N T E N T S Acknowledgments ..........................................................................................................xv Introduction ....................................................................................................................1 Chapter 1: Flow Fundamentals .........................................................................................9 Chapter 2: Collectors and Sensors ..................................................................................21 Chapter 3: Viewing Flows ..............................................................................................41 Chapter 4: Filtering Flows ..............................................................................................57 Chapter 5: Reporting and Follow-up Analysis ...................................................................81 Chapter 6: Perl, FlowScan, and Cflow.pm......................................................................117 Chapter 7: FlowViewer ................................................................................................139 Chapter 8: Ad Hoc Flow Visualization...........................................................................157 Chapter 9: Edges and Analysis.....................................................................................177 Index .........................................................................................................................191

Page 10

(This page has no text content)

Page 11

C O N T E N T S I N D E T A I L ACKNOWLEDGMENTS xv INTRODUCTION 1 Network Administration and Network Management ..................................................... 3 Network Management Tools ...................................................................................... 3 MRTG, Cricket, and Cacti ............................................................................ 3 RTG ........................................................................................................... 4 Nagios and Big Brother ............................................................................... 4 CiscoWorks, OpenView, and More .............................................................. 4 Enough Griping: What’s the Solution? ........................................................................ 5 Flow-Tools and Its Prerequisites .................................................................................. 6 Flows and This Book ................................................................................................. 6 1 FLOW FUNDAMENTALS 9 What Is a Flow? ..................................................................................................... 10 Flow System Architecture ......................................................................................... 11 The History of Network Flow .................................................................................... 12 NetFlow Versions ...................................................................................... 12 NetFlow Competition ................................................................................. 13 The Latest Standards .................................................................................. 13 Flows in the Real World .......................................................................................... 14 ICMP Flows .............................................................................................. 14 UDP Flows ................................................................................................ 15 TCP Flows ................................................................................................ 16 Other Protocols ......................................................................................... 17 Flow Export and Timeouts ........................................................................................ 18 Packet-Sampled Flows ............................................................................................. 19 2 COLLECTORS AND SENSORS 21 Collector Considerations ......................................................................................... 21 Operating System ..................................................................................... 22 System Resources ...................................................................................... 22 Sensor Considerations ............................................................................................ 22 Location ................................................................................................... 23 From Remote Facilities ............................................................................... 24 From Private Network Segments/DMZs ........................................................ 24 Implementing the Collector ...................................................................................... 24

Page 12

x Contents in Detai l Installing Flow-tools ................................................................................................. 25 Installing from Packages ............................................................................. 25 Installing from Source ................................................................................ 25 Running flow-capture .............................................................................................. 26 Starting flow-capture at Boot ....................................................................... 27 How Many Collectors? ........................................................................................... 28 Collector Log Files .................................................................................................. 28 Collector Troubleshooting ........................................................................................ 29 Configuring Hardware Flow Sensors ........................................................................ 29 Cisco Routers ............................................................................................ 30 Cisco Switches .......................................................................................... 30 Juniper Routers .......................................................................................... 31 Configuring Software Flow Sensors .......................................................................... 32 Setting Up Sensor Server Hardware ............................................................ 32 Network Setup .......................................................................................... 33 Sensor Server Setup ................................................................................... 34 Running the Sensor on the Collector ............................................................ 34 The Sensor: softflowd .............................................................................................. 34 Running softflowd ...................................................................................... 35 Watching softflowd ................................................................................... 35 3 VIEWING FLOWS 41 Using flow-print ...................................................................................................... 41 Printing Protocol and Port Names ................................................................ 43 Common Protocol and Port Number Assignments .......................................... 44 Viewing Flow Record Header Information with -p .......................................... 45 Printing to a Wide Terminal ........................................................................ 45 Setting flow-print Formats with -f ............................................................................... 46 Showing Interfaces and Ports in Hex with Format -f 0 ..................................... 46 Two Lines with Times, Flags, and Hex Ports Using -f 1 .................................... 47 Printing BGP Information ............................................................................ 48 Wide-Screen Display ................................................................................. 48 IP Accounting Format ................................................................................. 49 TCP Control Bits and Flow Records ........................................................................... 50 ICMP Types and Codes and Flow Records ................................................................ 52 Types and Codes in ICMP .......................................................................... 53 Flows and ICMP Details ............................................................................. 54 4 FILTERING FLOWS 57 Filter Fundamentals ................................................................................................. 58 Common Primitives .................................................................................... 58 Creating a Simple Filter with Conditions and Primitives .................................. 60 Using Your Filter ....................................................................................... 61

Page 13

Contents in Detai l xi Useful Primitives ..................................................................................................... 61 Protocol, Port, and Control Bit Primitives ....................................................... 61 IP Address and Subnet Primitives ................................................................. 64 Time, Counter, and Double Primitives ........................................................... 65 Interface and BGP Primitives ....................................................................... 67 Filter Match Statements ........................................................................................... 70 Protocols, Ports, and Control Bits ................................................................. 70 Addresses and Subnets .............................................................................. 72 Filtering by Sensor or Exporter .................................................................... 72 Time Filters ............................................................................................... 73 Clipping Levels ......................................................................................... 73 BGP and Routing Filters .............................................................................. 74 Using Multiple Filters .............................................................................................. 75 Logical Operators in Filter Definitions ....................................................................... 76 Logical “or” .............................................................................................. 76 Filter Inversion .......................................................................................... 77 Filters and Variables ............................................................................................... 78 Using Variable-Driven Filters ....................................................................... 79 Defining Your Own Variable-Driven Filters .................................................... 79 Creating Your Own Variables ..................................................................... 80 5 REPORTING AND FOLLOW-UP ANALYSIS 81 Default Report ........................................................................................................ 82 Timing and Totals ...................................................................................... 83 Packet Size Distribution .............................................................................. 84 Packets per Flow ....................................................................................... 84 Octets in Each Flow ................................................................................... 84 Flow Time Distribution ................................................................................ 85 Modifying the Default Report ................................................................................... 85 Using Variables: Report Type ...................................................................... 86 Using Variables: SORT .............................................................................. 86 Analyzing Individual Flows from Reports ................................................................... 88 Other Report Customizations ................................................................................... 89 Choosing Fields ........................................................................................ 89 Displaying Headers, Hostnames, and Percentages ........................................ 90 Presenting Reports in HTML ........................................................................ 91 Useful Report Types ................................................................................................ 92 IP Address Reports ..................................................................................... 92 Network Protocol and Port Reports .............................................................. 94 Traffic Size Reports .................................................................................... 96 Traffic Speed Reports ................................................................................. 97 Routing, Interfaces, and Next Hops ............................................................. 99 Reporting Sensor Output .......................................................................... 104 BGP Reports ........................................................................................... 104 Customizing Reports ............................................................................................. 107 Custom Report: Reset-Only Flows ............................................................... 107 More Report Customizations ..................................................................... 110 Customizing Report Appearance ............................................................... 112

Page 14

xii Contents in Detai l 6 PERL, FLOWSCAN, AND CFLOW.PM 117 Installing Cflow.pm ............................................................................................... 118 Testing Cflow.pm .................................................................................... 118 Install from Operating System Package ...................................................... 118 Install from Source ................................................................................... 119 Installing from Source with a Big Hammer .................................................. 119 flowdumper and Full Flow Information .................................................................... 119 FlowScan and CUFlow .......................................................................................... 120 FlowScan Prerequisites .......................................................................................... 121 Installing FlowScan and CUFlow ............................................................................ 121 FlowScan User, Group, and Data Directories .............................................. 122 FlowScan Startup Script ........................................................................... 123 Configuring FlowScan ............................................................................. 123 Configuring CUFlow: CUFlow.cf ............................................................... 124 Rotation Programs and flow-capture .......................................................... 127 Running FlowScan ................................................................................... 128 FlowScan File Handling ........................................................................... 128 Displaying CUFlow Graphs ...................................................................... 129 Flow Record Splitting and CUFlow ......................................................................... 130 Splitting Flows ........................................................................................ 131 Scripting Flow Record Splitting .................................................................. 132 Filtered CUFlow and Directory Setup ......................................................... 132 Using Cflow.pm ................................................................................................... 133 A Sample Cflow.pm Script ....................................................................... 133 Cflow.pm Variables ................................................................................. 134 Other Cflow.pm Exports ........................................................................... 135 Acting on Every File ................................................................................. 137 Return Value ........................................................................................... 137 Verbose Mode ........................................................................................ 138 7 FLOWVIEWER 139 FlowTracker and FlowGrapher vs. CUFlow .............................................................. 140 FlowViewer Security ............................................................................................. 140 Installing FlowViewer ............................................................................................ 140 Prerequisites ........................................................................................... 141 FlowViewer Installation Process ................................................................. 141 Configuring FlowViewer ........................................................................................ 141 Directories and Site Paths ......................................................................... 142 Website Setup ........................................................................................ 144 Devices and Exporters ............................................................................. 144 Troubleshooting the FlowViewer Suite ........................................................ 145 Using FlowViewer ................................................................................................ 146 Filtering Flows with FlowViewer ................................................................ 146 Reporting Parameters ............................................................................... 147 Printed Reports ........................................................................................ 149 Statistics Reports ..................................................................................... 149

Page 15

Contents in Detai l xiii FlowGrapher ....................................................................................................... 150 FlowGrapher Settings .............................................................................. 150 FlowGrapher Output ................................................................................ 151 FlowTracker ......................................................................................................... 152 FlowTracker Processes ............................................................................. 152 FlowTracker Settings ................................................................................ 152 Viewing Trackers ..................................................................................... 153 Group Trackers ....................................................................................... 154 Interface Names and FlowViewer ........................................................................... 156 8 AD HOC FLOW VISUALIZATION 157 gnuplot 101 ........................................................................................................ 158 Starting gnuplot ...................................................................................... 158 gnuplot Configuration Files ....................................................................... 159 Time-Series Example: Bandwidth ............................................................................ 160 Total Bandwidth Report ............................................................................ 160 Unidirectional Bandwidth Reports .............................................................. 168 Combined Inbound/Outbound Traffic ........................................................ 170 Automating Graph Production ............................................................................... 173 Comparison Graphs ............................................................................................. 175 Data Normalizing ................................................................................... 175 Time Scale ............................................................................................. 175 9 EDGES AND ANALYSIS 177 NetFlow v9 ......................................................................................................... 177 Installing flowd ....................................................................................... 178 Configuring flowd ................................................................................... 178 Converting flowd Data to Flow-tools .......................................................... 179 sFlow ................................................................................................................. 180 Configuring sFlow Export with sflowenable ................................................. 181 Convert sFlow to NetFlow ........................................................................ 181 Problem Solving with Flow Data ............................................................................. 182 Finding Busted Software ........................................................................... 182 Identifying Worms ................................................................................... 186 Traffic to Illegal Addresses ........................................................................ 187 Traffic to Nonexistent Hosts ...................................................................... 188 Afterword ............................................................................................................ 189 INDEX 191

Page 16

(This page has no text content)

Page 17

A C K N O W L E D G M E N T S Thanks to all the folks who have attended my net- work flow analysis tutorial over the years, and whose questions, comments, and ideas motivated this book. Now that the book is out, I’ll have to find something else to teach. And a special thanks to Mike O’Connor, who helped with the manuscript when he really should have been doing other things.

Page 18

(This page has no text content)

Page 19

I N T R O D U C T I O N Network administrators of all backgrounds share one underlying, overwhelming desire. It doesn’t matter if you manage a network with 400 separate manufacturing plants connected by a global MPLS mesh or if you’re responsible for three computers and an elderly printer. Network administra- tors all share an abiding and passionate desire for just one thing: We want our users to shut up. Blaming the network is easy. The network touches everything. Businesses assume that the network will work perfectly and make decisions accordingly. A user can’t open that 900MB Excel spreadsheet on the file server on another continent from his 20th-century PC? Network problem. A website in Faraway- istan is slow? Network problem. A user can’t get a faster response over a 33.6Kbps modem? Network problem. In general, users don’t care about triv- ialities such as the cost of bandwidth, the physical layout of transcontinental fiber, or the speed of light. They want the network to work the way they think it should.

Page 20

2 In t roduct ion This problem is exacerbated by the network’s invisibility. Routers and switches are black boxes. You feed cables to the network, and it gives connec- tivity back. Traditionally, servers have much better activity logging and report- ing than network devices. Sysadmins can write scripts to announce when a server process fails and try to remedy the situation, but very few network administrators have equipment that can monitor or repair itself. The usage information stored on a network device shows only the last few minutes at best. When a user reports an application failure, many systems administrators check their logs and report, “I have no problems. It must be the network.” More sophisticated network administrators have traffic measurement tools such as MRTG, CiscoWorks, or OpenView, but these don’t prove a network has no problems. They merely show whether the network has or lacks the particular set of problems that the software can report. The lack of media errors on a switch’s uplink port says nothing about TCP/IP errors or firewall issues. With such minimal logging, blaming the network becomes very easy and hard to disprove. In addition, modern-day network administrators typically have no formal certification or training process. The days of needing a computer science degree to manage a network are long gone. What network administration certifications exist are frequently for systems administration. An “operating system vendor–certified” network administrator is actually certified in the services that the operating system offers to the network, not the network itself. When you study for a Microsoft DHCP Server certification, for example, you learn some things about networking but a whole lot more about that par- ticular DHCP implementation. Network service management is a useful skill, mind you, but it’s not the same as managing the lowest level of the network. Network administrators learn the hard way—and it’s a very hard way indeed. This is a shame, because network administrators can claim a powerful role in any organization. The network administrator can help resolve almost every technical issue. The network administrator can make himself an invalu- able, indispensable, and irreplaceable component of the organization. A net- work administrator who masters the tools, understands the protocols, and restrains his natural contempt for the lesser minds surrounding him will not be fired until the company dissolves around him. Some network administrators understand this and are always ready to help. If a user or sysadmin reports an issue, this network admin is always ready to leap in with a packet analyzer or firewall log and offer insights. This is great customer service, if the user can replicate the problem at the time the network administrator is watching. Replicating problems consumes endless time and makes everyone involved testy. What if you had a tool that could tell you specifically what happened on the network yesterday or last week or last year? Imagine being able to pre- cisely identify the network impact of server and hardware changes. Suppose you could tell systems administrators exactly what bogus traffic their new server transmitted? Just for a moment, dream about being able to categori- cally and without risk of repudiation state, “That problem was not the net- work.” Then think about doing all that by taking advantage of your existing equipment.