Android System Programming Porting, customizing, and debugging Android HAL (Roger Ye) (z-library.sk, 1lib.sk, z-lib.sk)

Android System Programming Porting, customizing, and debugging Android HAL Roger Ye BIRMINGHAM - MUMBAI

Android System Programming Copyright © 2017 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: May 2017 Production reference: 1290517 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78712-536-0 www.packtpub.com

Credits Author Roger Ye Copy Editor Safis Editing Reviewers Bin Chen Chih-Wei Huang Shen Liu Nanik Tolaram Project Coordinator Ritika Manoj Commissioning Editor Amarabha Banerjee Proofreader Safis Editing Acquisition Editor Shweta Pant Indexer Mariammal Chettiyar Content Development Editor Arun Nadar Production Coordinator Nilesh Mohite Technical Editor Prajakta Mhatre

About the Author Roger Ye has worked in the area of embedded system programming for more than 10 years. He has worked on system programming for mobile devices, home gateways, and telecommunication systems for various companies, such as Motorola, Emerson, and Intersil. Most recently, he has worked as an engineering manager, leading a team of Android engineers to develop mobile security applications at Intel Security. With extensive knowledge and experience in the areas of embedded systems and mobile device development, he published a book called Embedded Programming with Android, Addison- Wesley, in 2015. I would like to thank my dearest wife, Bo Quan, and my lovely daughter, Yuxin Ye, for enduring me to spend significant time on this book over the weekends. They have been very encouraging and always give me support to work on the things that I am interested in.

About the Reviewers Bin Chen is a senior engineer from Linaro. He has worked on various Android-based products since 2010: TV, STB, Galaxy Tab, Nexus Player, and Google Project Ara, in that order, and now AOSP 96Boards. He occasionally blogs and speaks about all things Android. He lives in Sydney, Australia. Chih-Wei Huang is a developer and promoter of free software who lives in Taiwan. He is famous for his work in the VoIP and internationalization and localization fields in Greater China. Huang graduated from National Taiwan University (NTU) in 1993 with a bachelor's degree in physics, and attained a master's degree in the electrical engineering department of NTU in 2000. Huang currently works as a chief engineer of Tsinghua Tongfang Co., Ltd. for the OPENTHOS project. He is one of the founding members of the Software Liberty Association of Taiwan (SLAT). Chih-Wei Huang is the founder and coordinator of the Chinese Linux Documentation Project (CLDP). He is also the second coordinator of the Chinese Linux Extensions (CLE) and a core developer of GNU Gatekeeper (from 2001 to 2003). He is a contributor to pyDict, OpenH323, Asterisk, GStreamer, and more. He is working on a way to leverage the ASUS Eee PC with the power of the free software community and aims to provide a complete solution for Android on the x86 platform. The Eee PC, VirtualBox, and QEMU have been tested and are OK. Chih-Wei Huang and Yi Sun started the Android-x86 open source project in 2009. The project aims to bring Android to the x86 platform.

About the Reviewers Shen Liu is a senior engineer, working at Intel China. He used to work at McAfee LLC, Broadcom Corporation, and Huawei Technologies. He has over 10 years of work experience on Linux/Android and embedded systems, in different roles. He had taken manager, architect, and engineer roles during his career. He is mainly responsible for the Android framework, but is not limited to it, and he has a lot of passion for software design. On top of that, he loves reading technical books. Nanik Tolaram works as a senior Android platform engineer for BlocksGlobal in Australia, where he is responsible for developing Screener (screener.digital) and Lumin (mylumin.org). He is passionate about Android and is very active within both the local and international Android developer communities--from talks and teaching to writing articles for ODROID open source magazine (magazine.odroid.com). In his spare time, he loves to tinker with electronics and study human psychology and behavior. He lives in Sydney, Australia, with his lovely wife and two beautiful boys.

www.PacktPub.com For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. h t t p s ://w w w . p a c k t p u b . c o m /m a p t Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career. Why subscribe? Fully searchable across every book published by Packt Copy and paste, print, and bookmark content On demand and accessible via a web browser

Customer Feedback Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at www.amazon.com/dp/178712536X. If you'd like to join our team of regular reviewers, you can e-mail us at customerreviews@packtpub.com. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

Table of Contents Preface 1 Chapter 1: Introduction to Android System Programming 9 What is system programming? 10 What is the scope of this book? 11 Overview of the Android system 13 Kernel 13 HAL 15 Android system services 15 Binder IPC 16 Application framework 16 Recovery 17 The third-party open source projects derived from AOSP 19 LineageOS (CyanogenMod) 19 Android-x86 21 CWM/CMR/TWRP 22 Strategy of integration 23 Virtual hardware reference platforms 24 Introduction to the x86-based Android emulator 24 Introduction to ranchu 25 VirtualBox-based Android emulators 26 Summary 27 Chapter 2: Setting Up the Development Environment 28 Summary of Android versions 28 Installing Android SDK and setting up an Android Virtual Device 31 Creating AVD in an older version of SDK 31 Creating AVD in the latest version of SDK 33 Testing the goldfish emulator 34 Testing ranchu emulator 37 The AOSP build environment and the Android emulator build 39 The AOSP build environment 39 Installing the required packages 39 Installing Open JDK 7 and 8 39 Downloading the AOSP source 41 Installing repo 41

[ ii ] Initializing a repo client and downloading the AOSP source tree 41 Building AOSP Android emulator images 42 Testing AOSP images 45 Creating your own repository mirror 47 Repo and manifest 48 Using a local mirror for AOSP 50 Creating your own mirror of GitHub 52 Fetching Git repositories outside GitHub 54 Creating your own manifest for client download 57 Summary 60 Chapter 3: Discovering Kernel, HAL, and Virtual Hardware 61 What is inside the AOSP? 62 Android emulator HAL 63 Calling sequence 65 Goldfish lights HAL 67 The system service and hardware manager 69 Android emulator kernel and hardware 75 Android emulator hardware 77 Goldfish platform bus 77 QEMU pipe device 78 Goldfish audio device 79 Goldfish serial port 80 Goldfish kernel 81 QEMU pipe 82 Summary 84 Chapter 4: Customizing the Android Emulator 85 Why customize the Android emulator 85 Understanding build layers 86 Build variants 88 Creating a new x86emu device 89 Checking out from the AOSP 89 Checking out from a local mirror 90 Creating x86emu device 93 AndroidProducts.mk 94 BoardConfig.mk 97 device.mk 99 Building and testing x86emu 101 Building x86emu 101 Testing x86emu 105

[ iii ] Integrating with Eclipse 106 Summary 111 Chapter 5: Enabling the ARM Translator and Introducing Native Bridge 113 Introducing Native Bridge 113 Setting up Native Bridge as part of the ART initialization 116 Pre-initializing Native Bridge 123 Initializing Native Bridge 127 Loading a native library 130 Integrating Houdini to the x86emu device 134 Changing the configuration of the x86emu build 134 Extending the x86emu device 136 Changes to BoardConfig.mk 137 Changes to x86emu_x86.mk 138 Changes to device.mk 138 Using the Android-x86 implementation 139 Analyzing libnb.so 140 Using binfmt_misc 143 Building and testing 146 Testing the command-line application 147 Testing the Android JNI application 147 Summary 150 Chapter 6: Debugging the Boot Up Process Using a Customized ramdisk 151 Analyzing the Android start up process 152 Bootloader and the kernel 152 Analyzing the init process and ramdisk 153 Actions 155 Services 156 Device-specific actions and services 156 Source code and manifest changes 158 The Android-x86 start up process 160 The first-stage boot using initrd.img 160 Inside initrd.img 161 Inside install.img 166 Building x86emu with initrd.img 168 Creating a filesystem image 170 Kernel changes 173 Booting a disk image on the Android emulator 174 Summary 179 Chapter 7: Enabling Wi-Fi on the Android Emulator 180

[ iv ] Wi-Fi on Android 180 The Wi-Fi architecture 181 QEMU networking and wpa_supplicant in Android 190 Adding Wi-Fi to the emulator 191 Enabling wpa_supplicant in BoardConfig.mk 191 Providing a proper wpa_supplicant configuration 192 Creating services in init scripts 194 Initializing network interface eth1 194 Starting up wpa_supplicant 195 Building the source code 196 Getting the source code 196 Enabling boot with initrd.img 198 Testing Wi-Fi on an emulator 201 Booting an Android emulator using initrd.img 201 Booting an Android emulator using ramdisk.img 203 Debugging Wi-Fi start up processes 203 Summary 206 Chapter 8: Creating Your Own Device on VirtualBox 207 HAL of x86vbox 208 The manifest for x86vbox 210 Creating a new x86vbox device 214 Product definition Makefile of x86vbox 214 Board configuration of x86vbox 215 Common x86 devices 216 Getting the source code and building the x86vbox device 220 Boot up process and device initialization 220 Device initialization before Android start-up 221 HAL initialization during the Android start-up 228 Summary 232 Chapter 9: Booting Up x86vbox Using PXE/NFS 233 Setting up a PXE boot environment 234 Preparing PXE Boot ROM 235 Downloading and building the LAN Boot ROM 236 Fixing up the ROM image 236 Configuring the virtual machine to use the LAN Boot ROM 237 Setting up the PXE boot environment 239 Configuring and testing the PXE boot 239 Setting up the virtual machine 240 Using VirtualBox internal PXE booting with NAT 242

[ v ] Configuring pxelinux.cfg 243 pxelinux.cfg/default 244 Setting up a serial port for debugging 246 NFS filesystem 248 Preparing the kernel 249 Setting up the NFS server 252 Configuring the PXE boot menu 252 Booting to NFS installation 253 Booting from a hard disk 255 Booting to recovery 255 Summary 258 Chapter 10: Enabling Graphics 259 Introduction to the Android graphics architecture 260 Delving into graphics HAL 262 Loading the Gralloc module 262 Initializing GPU 269 Initializing framebuffer 271 Allocating and releasing the graphic buffer 280 Allocating from framebuffer 285 Allocating from system memory 287 Releasing graphic buffers 289 Rendering framebuffer 291 Graphics HAL of the Android emulator 294 Overview of hardware GLES emulation 295 Initializing GPU0 and FB0 in GLES emulation 299 GPU0 device implementation 302 FB0 device implementation 307 Summary 308 Chapter 11: Enabling VirtualBox-Specific Hardware Interfaces 309 OpenGL ES and graphics hardware initialization 309 Loading OpenGL ES libraries 310 Analyzing the loading process 313 Loading the driver 318 Creating the rendering engine 320 The uvesafb framebuffer driver 322 What is uvesafb? 322 Testing the uvesafb framebuffer driver 322 Initializing uvesafb in x86vbox 328 Integrating VirtualBox Guest Additions 331 Building VirtualBox Guest Additions 333

[ vi ] Integrating vboxsf 334 Integrating vboxvideo 337 Building and testing images with VirtualBox Guest Additions 339 Summary 339 Chapter 12: Introducing Recovery 340 Recovery introduction 340 Android device partitions 341 Analyzing recovery 343 BCB 344 Cache partition 347 Main flow of recovery 348 Retrieving arguments from BCB and cache files 352 Factory data reset 356 OTA update 358 Building recovery for x86vbox 364 Building configuration 364 Changes to x86vbox 365 Changes to recovery 368 Changes to newinstaller 369 Testing recovery 371 Summary 371 Chapter 13: Creating OTA Packages 372 What is inside an OTA package 372 Updater 375 The updater script 380 Edify functions 380 Built-in functions 380 Installation functions 381 Block image functions 385 Device extensions 385 Preparing an OTA package for x86vbox 386 Removing dependencies on /system 388 Hardware initialization in recovery 389 Minimum execution environment in recovery 391 Building and testing 394 Summary 397 Chapter 14: Customizing and Debugging Recovery 398 Debugging and testing native Android applications 399

[ vii ] Debugging with GDB 400 Integration with Eclipse 403 Extending recovery and the updater 409 Extending recovery 409 Extending the updater 416 Extending the Python module 424 Building and testing the extended recovery and updater 425 Supporting the third-party recovery packages 433 Xposed recovery package 433 Opening GApps 437 Summary 441 Index 442

Preface Android is the most popular mobile operating system in the world. Since 2013, Android has around 80% market share worldwide, while the second largest mobile operating system, iOS, has less than 20% market share. Due to the popularity of Android, there are many books about Android programming in the market. Most of them are targeted at Android application developers, which are the largest community in the world of Android development. There is also another group of people working on the layer beneath the Android framework. Many people call them Android system developers. Comparing to Android application developers, Android system developers use the C/C++ languages, or even assembly language, to develop system services or device drivers. The scope and the definition of Android system development is much more vague than comparing to Android application development. For Android application development, the development environment and tools are very clear: the Android SDK and Android Studio from Google should be used and the programming language is Java. For Android system development, we may use the Android NDK to develop Android system services or native applications. Many people refer to development based on the Android Open Source Project (AOSP) as Android system development. Nevertheless, Android system development encompasses the activities that produces native applications, services, or device drivers for a particular hardware platform. It closer to the hardware and the operating system, whereas Android application development is more general and hardware-independent. Due to the hardware and operating system dependencies, it is more difficult to teach Android system programming than Android application programming. From the number of books in the market, we can see this. It is much easier to teach Android application development using specific examples. The readers of application programming books can follow the examples and can test them on most available Android devices. However, most Android system programming book can only talk about general concepts or ideas. When the authors want to use examples, they must pertain to a particular hardware platform and Android version. This makes it difficult for readers to repeat the same process.

Preface [ 2 ] Virtual hardware platforms To make the discussion more general and overcome the issue of specific hardware platforms, I use virtual hardware platforms to demonstrate the work at the Android system level. Before this book, I made an attempt to use a virtual hardware platform to explain how we can learn embedded system programming using an Android emulator in my previous book, Embedded Programming with Android. It seems many readers liked the idea, because they can explore the code examples much more easily on a virtual hardware platform that is available for everyone. Android version used in this book Android is still changing at a very fast pace. When I completed the book Embedded Programming with Android, we were still using Android 5 (Lollipop), and Android 6 (Marshmallow) was on the way to market with preview releases. Now while I am working on this book, Android 7 devices are available on the market and the next release of Android 8 has been announced with preview releases. We will use Android 7 (Nougat) to build all source code used in this book. What this book covers In this book, we discuss the Android system programming practices. We will use two projects (x86emu and x86vbox) to teach essential knowledge of Android system programming. The book is split into includes two parts. The first part of this book talks about how to customize, extend, and port an Android system. We will use an Android emulator as the virtual hardware platform to demonstrate how to customize and extend an Android system. You will learn how to integrate an ARM translator (Houdini) into the Intel x86-based emulator and how to add Wi-Fi support to an Android emulator. We will use an x86emu device to learn these topics. After that, we will learn how to port an Android system to a new platform using VirtualBox. You will learn how to boot Android in the PXE/NFS environment, how to enable the graphics system, and how to integrate VirtualBox Guest Additions into the Android system. We will use x86vbox device to learn these topics.

Preface [ 3 ] In the second part of this book, we will learn how to update or patch a released system using recovery. In this part, we will provide a general introduction to recovery first. After that, we will explore how to build recovery for x86vbox device. With recovery for x86vbox device, we will demonstrate how to flash an update package to change the system image. We will use examples such as the Gapps and xposed recovery packages to demonstrate how to update an Android system image using third-party recovery packages. Chapter 1, Introduction to Android System Programming, covers a general introduction of Android system programming. It also explains the scope of this book. Chapter 2, Setting Up the Development Environment, provides details of the development environment setup for AOSP programming. After we set up the development environment, we will build an Android emulator image to test our setup. Other than the environment setup, we specifically discuss how to create your own source code mirror of AOSP from GitHub to help your quickly switch between different configurations. Chapter 3, Discovering Kernel, HAL, and Virtual Hardware, covers an introduction to the Linux kernel, Hardware Abstraction Layer, and virtual hardware. In this chapter, we look at all the layers in the Android system software stack related to porting. We also take a in- depth look at the internals of the virtual hardware that we are going to use in this book. Chapter 4, Customizing the Android Emulator, covers the development of a new device, x86emu. We will learn how to customize and extend this device in the next few chapters. Chapter 5, Enabling the ARM Translator and Introducing Native Bridge, explores a new feature introduced in Android 5--Native Bridge. Since we created an x86-based device, x86emu, we have to integrate the ARM translator module (Houdini) into our device so that most ARM- native applications can run on it. Chapter 6, Debugging the Boot Up Process Using a Customized ramdisk, introduces an advanced debugging skill to troubleshoot issues during the boot up stage. The famous Android-x86 project uses a special ramdisk to start the boot up process. It helps to troubleshoot device driver and init process issues very easily. Chapter 7, Enabling Wi-Fi on the Android Emulator, presents details of how to enable Wi-Fi on our Android emulator. The Android emulator only supports an emulated 3G data connection, but many applications are aware of data and the Wi-Fi connection. We demonstrate how to enable Wi-Fi in the Android emulator in this chapter.

Preface [ 4 ] Chapter 8, Creating Your Own Device on VirtualBox, explores how to port Android on VirtualBox by introducing a new device x86vbox. The x86emu device is used to demonstrate how to customize an existing implementation, while x86vbox is used to demonstrate how to port Android to a new hardware platform. Chapter 9, Booting Up x86vbox Using PXE/NFS, explains how to boot up Android on VirtualBox using PXE/NFS. Since VirtualBox is a general virtual hardware, the first problem that we meet is we need a bootloader to boot the system. We will use the PXE/NFS boot to solve this issue. This is an advanced debugging skills which can be used in your own project. To discuss a more advanced case about the PXE/NFS setup using an external DHCP/TFTP server running in the host-only network environment, I have written an article, which you can find at h t t p s ://w w w . p a c k t p u b . c o m /b o o k s /c o n t e n t /b o o t i n g - a n d r o i d - s y s t e m - u s i n g - p x e n f s . Chapter 10, Enabling Graphics, covers the Android graphic system. We introduce the Android graphics architecture and how to enable it on the x86vbox device. Chapter 11, Enabling VirtualBox-Specific Hardware Interfaces, explains how to integrate the device drivers in VirtualBox Guest Additions into the Android system. Chapter 12, Introducing Recovery, provides an introduction to recovery. We will learn how to customize and port recovery to a new hardware platform by building a recovery for the x86vbox device. Chapter 13, Creating OTA Packages, covers the scripting language used by recovery: Edify. We will learn how to build and test OTA updates. Chapter 14, Customizing and Debugging Recovery, expands on the concepts we learned about recovery and OTA packages. We will customize both recovery and updater for x86vbox device. We will test third-party OTA packages from Gapps and Xposed using our own recovery. What you need for this book To read this book, you should have essential knowledge of embedded operating systems and C/C++ programming language.