Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 1 | DexFuzz |
| 2 | ======= |
| 3 | |
| 4 | DexFuzz is primarily a tool for fuzzing DEX files. Fuzzing is the introduction of |
| 5 | subtle changes ("mutations") to a file to produce a new test case. These test cases |
| 6 | can be used to test the various modes of execution available to ART (Interpreter, |
Branislav Rankov | f559b57 | 2016-11-23 11:41:43 +0000 | [diff] [blame] | 7 | Optimizing compiler) to check for bugs in these modes of execution. |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 8 | This is done by differential testing - each test file is executed with each mode of |
| 9 | execution, and any differences between the resulting outputs may be an indication of |
| 10 | a bug in one of the modes. |
| 11 | |
| 12 | For a wider overview of DexFuzz, see: |
| 13 | |
| 14 | http://community.arm.com/groups/android-community/blog/2014/11/26/the-art-of-fuzz-testing |
| 15 | |
| 16 | In typical operation, you provide DexFuzz with a set of DEX files that are the "seeds" |
| 17 | for mutation - e.g. some tests taken from the ART test suite - and point it at an |
| 18 | ADB-connected Android device, and it will fuzz these seed files, and execute the |
| 19 | resulting new tests on the Android device. |
| 20 | |
| 21 | How to run DexFuzz |
| 22 | ================== |
| 23 | |
Stephen Kyle | 807f178 | 2015-03-24 17:44:27 +0000 | [diff] [blame] | 24 | DexFuzz can run its test programs on either an ADB-connected device, or a host-build of |
| 25 | ART locally. |
| 26 | |
| 27 | Execution on an ADB-connected device |
| 28 | ------------------------------------ |
| 29 | |
| 30 | 1. Build dexfuzz with mmma tools/dexfuzz from within art/. |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 31 | 2. Make sure you have an Android device connected via ADB, that is capable of |
| 32 | having DEX files pushed to it and executed with the dalvikvm command. |
| 33 | 3. Make sure you're in the Android build environment! |
| 34 | (That is, . build/envsetup.sh && lunch) |
| 35 | 4. Create a new directory, and place some DEX files in here. These are the seed files |
| 36 | that are mutated to form new tests. |
| 37 | 5. Create a directory on your device that mutated test files can be pushed to and |
| 38 | executed in, using dalvikvm. For example, /data/art-test/ |
| 39 | 6. If you currently have multiple devices connected via ADB, find out the name of |
| 40 | your device using "adb devices -l". |
| 41 | 7. Run this command: |
| 42 | |
| 43 | dexfuzz --inputs=<seeds dir> --execute --repeat=<attempts> \ |
| 44 | --dump-output <combination of ISA(s) and and backend(s)> |
| 45 | |
| 46 | You MUST specify one of the following ISAs: |
| 47 | --arm |
| 48 | --arm64 |
| 49 | --x86 |
| 50 | --x86_64 |
| 51 | --mips |
| 52 | --mips64 |
| 53 | |
| 54 | And also at least two of the following backends: |
| 55 | --interpreter |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 56 | --optimizing |
| 57 | |
| 58 | Note that if you wanted to test both ARM and ARM64 on an ARM64 device, you can use |
| 59 | --allarm. Also in this case only one backend is needed, if i.e., you wanted to test |
Branislav Rankov | f559b57 | 2016-11-23 11:41:43 +0000 | [diff] [blame] | 60 | ARM Optimizing Backend vs. ARM64 Optimizing Backend. |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 61 | |
| 62 | Some legal examples: |
Branislav Rankov | f559b57 | 2016-11-23 11:41:43 +0000 | [diff] [blame] | 63 | --arm --optimizing --interpreter |
| 64 | --x86 --optimizing --interpreter |
| 65 | --allarm --optimizing |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 66 | |
| 67 | Add in --device=<device name, e.g. device:generic> if you want to specify a device. |
| 68 | Add in --execute-dir=<dir on device> if you want to specify an execution directory. |
| 69 | (The default is /data/art-test/) |
| 70 | |
Stephen Kyle | 807f178 | 2015-03-24 17:44:27 +0000 | [diff] [blame] | 71 | Host Execution |
| 72 | -------------- |
| 73 | |
| 74 | DexFuzz now supports execution on your host machine. |
| 75 | Follow steps 1, 3, 4, and 7 as above, but also observe the following: |
| 76 | - instead of specifying an ISA, use --host |
| 77 | - ANDROID_DATA must be set, pointing to a location where dex2oat will place |
| 78 | OAT files after compilation. |
| 79 | - Files will always be executed in the same directory where you are executing DexFuzz. |
| 80 | |
| 81 | Fuzzer Operation |
| 82 | ---------------- |
| 83 | |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 84 | As the fuzzer works, you'll see output like: |
| 85 | |
| 86 | |-----------------------------------------------------------------| |
| 87 | |Iterations|VerifyFail|MutateFail|Timed Out |Successful|Divergence| |
| 88 | |-----------------------------------------------------------------| |
| 89 | | 48 | 37 | 4 | 0 | 6 | 1 | |
| 90 | |
| 91 | Iterations - number of attempts we've made to mutate DEX files. |
| 92 | VerifyFail - the number of mutated files that ended up failing to verify, either |
| 93 | on the host, or the target. |
| 94 | MutateFail - because mutation is a random process, and has attempt thresholds to |
| 95 | avoid attempting to mutate a file indefinitely, it is possible that |
| 96 | an attempt to mutate a file doesn't actually mutate it. This counts |
| 97 | those occurrences. |
| 98 | Timed Out - mutated files that timed out for one or more backends. |
| 99 | Current timeouts are: |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 100 | Optimizing - 5 seconds |
Aart Bik | 5618a57 | 2017-01-24 10:27:52 -0800 | [diff] [blame] | 101 | Interpreter - 30 seconds |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 102 | (use --short-timeouts to set all backends to 2 seconds.) |
| 103 | Successful - mutated files that executed and all backends agreed on the resulting |
| 104 | output. NB: if all backends crashed with the same output, this would |
| 105 | be considered a success - proper detection of crashes is still to come. |
| 106 | Divergence - mutated files that executed and some backend disagreed about the |
| 107 | resulting output. Divergent programs are run multiple times with a |
| 108 | single backend, to check if they diverge from themselves, and these are |
| 109 | not included in the count. If multiple architectures are being used |
| 110 | (ARM/ARM64), and the divergences align with different architectures, |
| 111 | these are also not included in the count. |
| 112 | |
| 113 | 8. Check report.log for the full report, including input file and RNG seed for each |
| 114 | test program. This allows you to recreate a bad program with, e.g.: |
| 115 | |
| 116 | dexfuzz --input=<input file> --seed=<seed value> |
| 117 | |
| 118 | Check dexfuzz --help for the full list of options. |
| 119 | |
| 120 | NOTE: DEX files with unicode strings are not fully supported yet, and DEX files with |
| 121 | JNI elements are not supported at all currently. |
| 122 | |
| 123 | Mutation Likelihoods |
| 124 | ==================== |
| 125 | |
| 126 | Each bytecode mutation has a chance out of 100% of firing. Following is the listing |
| 127 | of each mutation's probability. If you wish to easily adjust these values, copy |
| 128 | these values into a file called likelihoods.txt, and run dexfuzz with |
| 129 | --likelihoods=likelihoods.txt. |
| 130 | |
| 131 | ArithOpChanger 75 |
| 132 | BranchShifter 30 |
| 133 | CmpBiasChanger 30 |
| 134 | ConstantValueChanger 70 |
| 135 | ConversionRepeater 50 |
| 136 | FieldFlagChanger 40 |
| 137 | InstructionDeleter 40 |
| 138 | InstructionDuplicator 80 |
| 139 | InstructionSwapper 80 |
Sumnima Joshi | c7c3d85 | 2017-06-30 17:10:38 -0700 | [diff] [blame] | 140 | InvokeChanger 30 |
Sumnima Joshi | 9161721 | 2017-07-14 11:52:23 -0700 | [diff] [blame] | 141 | NewArrayLengthChanger 50 |
Sumnima Joshi | 9557d05 | 2017-07-26 16:31:42 -0700 | [diff] [blame] | 142 | NewInstanceChanger 10 |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 143 | NewMethodCaller 10 |
| 144 | NonsenseStringPrinter 10 |
Sumnima Joshi | abae637 | 2017-06-22 15:03:02 -0700 | [diff] [blame] | 145 | OppositeBranchChanger 40 |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 146 | PoolIndexChanger 30 |
Sumnima Joshi | abae637 | 2017-06-22 15:03:02 -0700 | [diff] [blame] | 147 | RandomBranchChanger 30 |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 148 | RandomInstructionGenerator 30 |
Sumnima Joshi | ad70831 | 2017-07-25 14:39:35 -0700 | [diff] [blame] | 149 | RegisterClobber 10 |
Stephen Kyle | 959ffdf | 2014-11-28 14:27:44 +0000 | [diff] [blame] | 150 | SwitchBranchShifter 30 |
| 151 | TryBlockShifter 40 |
| 152 | ValuePrinter 40 |
| 153 | VRegChanger 60 |