1# Dittosuite 2 3Dittosuite is a work in progress collection of tools that aims at providing 4a high-level language called Dittolang that defines operations. 5 6The defined Dittolang operations can be interpreted by Dittosim for simulation 7to provide a simulated performance measurement and quickly identify 8the goodness of a solution. 9 10Specularly, Dittobench interprets the Dittolang operations and executes them on 11a real device, tracking the behavior and measuring the performance. 12 13# Doxygen documentation 14 15In addition to this readme, the project includes a Doxygen configuration, which 16can be generated with the following command: 17 18``` 19$ doxygen 20``` 21 22# How to run 23 24``` 25$ ./dittobench [options] [.ditto file] 26``` 27 28To run a benchmark, a well formed .ditto file must be provided, see section 29[How to write .ditto files](#how-to-write-ditto-files) 30In addition, these options can be set: 31 32- `--results-output=<int | string>` (default: report). Select the results output format. 33Options: report, csv with 0, 1 respectively. 34- `--log-stream=<int | string>` (default: stdout). Select the output stream for the log messages. 35Options: stdout, logcat with 0, 1 respectively. 36- `--log-level=<int | string>` (default: INFO). Select to output messages which are at or below 37the set level. Options: VERBOSE, DEBUG, INFO, WARNING, ERROR, FATAL with 0, 1, 2, 3, 4 and 5 38respectively. 39- `--parameters=string`. If the benchmark is parametric, all the parameters (separated by commas) 40can be given through this option. 41 42# How to write .ditto files 43 44Every .ditto file should begin with this skeleton: 45``` 46main: { 47 ... 48}, 49global { 50 ... 51} 52``` 53 54Optionally, it can contain `init` and `clean_up` sections: 55``` 56init: { 57 ... 58}, 59main: { 60 ... 61}, 62clean_up: { 63 ... 64}, 65global { 66 ... 67} 68``` 69 70## `global` 71 72Global section should contain general benchmark configuration. Currently available options: 73 74- (optional) `string absolute_path` (`default = ""`). Specifies the absolute path for the files. 75 76## `init` 77 78`init` is optional and can be used to initialize the benchmarking environment. It executes 79instructions similar to `main`, but the results are not collected in the end. 80 81## `main` 82 83`main` is the entry point for the benchmark. It can contain a single `instruction` or 84`instruction_set` (also with nested `instruction_set`). 85 86## `clean_up` 87 88`clean_up` is optional and can be used to reset the benchmarking environment to the initial state, 89e.g, delete benchmark files. Similar to `init`, it executes instructions like `main`, but results 90are not collected in the end. 91 92## `instruction` 93 94``` 95{ 96 <name of the instruction>: { 97 <first argument>, 98 <second argument>, 99 ... 100 }, 101 <general instruction options> 102} 103``` 104 105Currently available options: 106- (optional) `int repeat` (`default = 1`). Specifies how many times the instruction should be 107repeated. 108 109## `instruction_set` 110``` 111{ 112 instruction_set: { 113 instructions: { 114 { 115 <name of the first instruction>: { 116 <first argument>, 117 <second argument>, 118 ... 119 }, 120 <general instruction options> 121 }, 122 { 123 <name of the second instruction>: { 124 <first argument>, 125 <second argument>, 126 ... 127 }, 128 <general instruction options> 129 }, 130 ... 131 }, 132 iterate_options: {...} 133 }, 134 <general instruction options> 135} 136``` 137 138Instruction set is an Instruction container that executes the contained instructions sequentially. 139Instruction set can optionally iterate over a list and execute the provided set of instructions on 140each item from the list. To use it, `iterate_options` should be set with these options: 141- `string list_name` - Shared variable name pointing to a list of values. 142- `string item_name` - Shared variable name to which a selected value should be stored. 143- (optional) `Order order` (`default = SEQUENTIAL`) - Specifies if 144 the elements of the list should be accessed sequentially or randomly. Options: 145 `SEQUENTIAL`, `RANDOM`. 146- (optional) `Reseeding reseeding` (`default = ONCE`) - Specifies how often the random number 147generator should be reseeded with the same provided (or generated) seed. Options: `ONCE`, 148`EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 149- (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, 150current system time is used as the seed. 151 152## `multithreading` and `threads` 153 154``` 155multithreading: { 156 threads: [ 157 { 158 instruction: {...}, 159 spawn: <number of threads to spawn with the provided instruction> 160 }, 161 ... 162 ] 163} 164``` 165 166Multithreading is another instruction container that executes the specified instructions 167(or instruction sets) in different threads. If the optional `spawn` option for a specific 168instruction (or instruction set) is provided, then the provided number of threads will be created 169for it. 170 171## Example 172 173``` 174main: { 175 instruction_set: { 176 instructions: [ 177 { 178 open_file: { 179 path_name: "newfile2.txt", 180 output_fd: "test_file" 181 } 182 }, 183 { 184 close_file: { 185 input_fd: "test_file" 186 } 187 } 188 ] 189 }, 190 repeat: 10 191}, 192global { 193 absolute_path: "/data/local/tmp/"; 194} 195``` 196See more examples in `example/`. 197 198# Predefined list of instructions 199 200## `open_file` 201 202Opens the file with a file path or a shared variable name pointing to a file path. If neither of 203those are provided, a random name consisting of 9 random digits is generated. Optionally saves the 204file descriptor which can then be used by subsequent instructions. Also, can optionally create the 205file if it does not already exist. 206 207### Arguments: 208- (optional) `string path_name` - Specifies the file path.<br/> 209OR<br/> 210`string input` - Shared variable name pointing to a file path. 211- (optional) `string output_fd` - Shared variable name to which output file descriptor 212should be saved. 213- (optional) `bool create` (`default = true`) - Specifies if the file should be created if it does 214not already exist. If the file exists, nothing happens. 215 216## `delete_file` 217 218Deletes the file with a file path or a shared variable name pointing to a file path. 219Uses `unlink(2)`. 220 221### Arguments: 222- `string path_name` - Specifies the file path.<br/> 223OR<br/> 224`string input` - Shared variable name pointing to a file path. 225 226 227## `close_file` 228 229Closes the file with the provided file descriptor. 230Uses `close(2)`. 231 232### Arguments: 233- `string input_fd` - Shared variable name pointing to a file descriptor. 234 235## `resize_file` 236 237Resizes the file with the provided file descriptor and new size. If the provided size is greater 238than the current file size, `fallocate(2)` is used, while `ftruncate(2)` is used if the provided 239size is not greater than the current file size. 240 241### Arguments: 242- `string input_fd` - Shared variable name pointing to a file descriptor. 243- `int64 size` - New file size (in bytes). 244 245## `resize_file_random` 246 247Resizes the file with the provided file descriptor and a range for new size. New file size is 248randomly generated in the provided range and if the generated size is greater 249than the current file size, `fallocate(2)` is used, while `ftruncate(2)` is used if the generated 250size is not greater than the current file size. 251 252### Arguments: 253- `string input_fd` - Shared variable name pointing to a file descriptor. 254- `int64 min` - Minimum value (in bytes) 255- `int64 max` - Maximum value (in bytes) 256- (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, 257current system time is used as the seed. 258- (optional) `Reseeding reseeding` (`default = ONCE`). How often the random number 259generator should be reseeded with the provided (or generated) seed. Options: `ONCE`, 260`EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 261 262## `write_file` 263 264Writes to file with the provided file descriptor. For `SEQUENTIAL` 265access, the blocks of data will be written sequentially and if the end of 266the file is reached, new blocks will start from the beginning of the file. For 267`RANDOM` access, the block offset, to which data should be written, will 268be randomly chosen with uniform distribution. `10101010` byte is used for the 269write operation to fill the memory with alternating ones and zeroes. Uses 270`pwrite64(2)`. 271 272### Arguments: 273- `string input_fd` - Shared variable name pointing to a file descriptor. 274- (optional) `int64 size` (`default = -1`) - How much data (in bytes) should be written in total. 275If it is set to `-1`, then file size is used. 276- (optional) `int64 block_size` (`default = 4096`) - How much data (in bytes) should be written at 277once. If it is set to `-1`, then file size is used. 278- (optional) `int64 starting_offset` (`default = 0`) - If `access_order` is 279 set to `SEQUENTIAL`, then the blocks, to which the data should be written, 280 will start from this starting offset (in bytes). 281- (optional) `Order access_order` (`default = SEQUENTIAL`) - Order of the 282 write. Options: `SEQUENTIAL` and `RANDOM`. 283- (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, 284current system time is used as the seed. 285- (optional) `bool fsync` (`default = false`) - If set, `fsync(2)` will be called after the 286execution of all write operations. 287- (optional) `Reseeding reseeding` (`default = ONCE`) - How often the random number 288generator should be reseeded with the provided (or generated) seed. Options: `ONCE`, 289`EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 290 291## `read_file` 292 293Reads from file with the provided file descriptor. For `SEQUENTIAL` 294access, the blocks of data will be read sequentially and if the end of 295the file is reached, new blocks will start from the beginning of the file. For 296`RANDOM` access, the block offset, from which data should be read, will 297be randomly chosen with uniform distribution. Calls `posix_fadvise(2)` before 298the read operations. Uses `pread64(2)`. 299 300### Arguments: 301- `string input_fd` - Shared variable name pointing to a file descriptor. 302- (optional) `int64 size` (`default = -1`) - How much data (in bytes) should be read in total. 303If it is set to `-1`, then file size is used. 304- (optional) `int64 block_size` (`default = 4096`) - How much data (in bytes) should be read at 305once. If it is set to `-1`, then file size is used. 306- (optional) `int64 starting_offset` (`default = 0`) - If `access_order` is 307 set to `SEQUENTIAL`, then the blocks, from which the data should be read, 308 will start from this starting offset (in bytes). 309- (optional) `Order access_order` (`default = SEQUENTIAL`) - Order of the 310 read. Options: `SEQUENTIAL` and `RANDOM`. 311- (optional) `uint32 seed` - Seed for the random number generator. If the seed is not provided, 312current system time is used as the seed. 313- (optional) `ReadFAdvise fadvise` (`default = AUTOMATIC`) - Sets the argument 314 for the `posix_fadvise(2)` operation. Options: `AUTOMATIC`, `NORMAL`, 315 `SEQUENTIAL` and `RANDOM`. If `AUTOMATIC` is set, then 316 `POSIX_FADV_SEQUENTIAL` or `POSIX_FADV_RANDOM` will be used for `SEQUENTIAL` 317 and `RANDOM` access order respectively. 318- (optional) `Reseeding reseeding` (`default = ONCE`) - How often the random number 319generator should be reseeded with the provided (or generated) seed. Options: `ONCE`, 320`EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 321 322## `read_directory` 323 324Reads file names from a directory and stores them as a list in a shared variable. Uses `readdir(3)`. 325 326### Arguments: 327- `string directory_name` - Name of the directory 328- `string output` - Shared variable name to which files names should be saved. 329 330## `invalidate_cache` 331 332Drops kernel clean caches, including, dentry, inode and page caches by calling sync() first and 333then writing `3` to `/proc/sys/vm/drop_caches`. No arguments. 334 335# Dependencies 336 337## Android 338 339The project is currently being developed as part of the Android Open Source 340Project (AOSP) and is supposed to run out-of-the-box. 341 342## Linux 343 344The following utilities are required to build the project on Linux: 345 346``` 347sudo apt install cmake protobuf-compiler 348 349``` 350 351# Testing 352 353## Linux 354 355A suite of unit tests is provided in the test/ directory. 356In Linux these tests can be run with the following commands: 357 358``` 359mkdir build 360cd build 361make 362cd test 363ctest 364``` 365 366### Coverage 367 368As an extension to testing, coverage is not a metric that guarantees good 369quality testing, but at least shows what is not been tested yet. 370 371One way of getting coverage data is to rely on `llvm` to build the code with 372extra flags to generate coverage information, `llvm-cov` to extract coverage 373data, and `lcov` to aggregate and export all the coverage information into a 374human-readable format. 375 376``` 377mkdir build 378cd build 379CC=/usr/bin/clang CXX=/usr/bin/clang++ cmake -DCMAKE_BUILD_TYPE=Debug .. 380make 381ctest --test-dir test 382lcov -d ./CMakeFiles/ -b . --gcov-tool $PWD/../test/llvm-gcov.sh --capture -o cov.info 383genhtml cov.info -o coverage_html 384``` 385 386> **_NOTE:_** lcov version 2.* has issues such as `geninfo: ERROR: "XXX: 387function YYY found on line but no corresponding 'line' coverage data point. 388Cannot derive function end line.`. This can be solved by downgrading to version 3891.6. The lcov repository already has a binary, so PATH can be updated with its 390`bin` folder. 391 392 393# Use cases 394 395 396## File operations performance (high priority) 397 398Bandwidth and measurement when dealing with few huge files or many small files. 399Operations are combinations of sequential/random-offset read/write. 400 401Latency in creating/deleting files/folders. 402 403These operations should be able to be triggered in a multiprogrammed fashion. 404 405 406## Display pipeline 407 408A graph of processes that are communicating with each others in a pipeline of 409operations that are parallely contributing to the generation of display frames. 410 411 412## Scheduling (low priority) 413 414Spawning tasks (period, duration, deadline) and verifying their scheduling 415latency and deadline misses count. 416 417 418# Workflow example and implementation nits 419 420In the following scenario, two threads are running. 421 422T1 runs the following operations: Read, Write, Read, sends a request to T2 and 423waits for the reply, then Write, Read. 424T2 waits for a request, then Read, Write, then sends the reply to the requester. 425 426Operations are encoded as primitives expressed with ProtoBuffers. 427The definition of dependencies among threads can be represented as graphs. 428 429 430## Initialization phase 431 432The first graph traversal is performed at initialization time, when all the 433ProtoBuffer configuration files are distributed among all the binaries so that 434they can perform all the heavy initialization duties. 435 436 437## Execution phase 438 439After the initialization phase completes the graph can be traversed again to 440put all the workloads in execution. 441 442 443## Results gathering phase 444 445A final graph traversal can be performed to fetch all the measurements that 446each entity internally stored. 447 448 449## Post processing 450 451All the measurements must be ordered and later processed to provide useful 452information to the user. 453 454T1: INIT : [ RD WR RD ] SND RCV [ WR RD ] : END 455T2: INIT : RCV [ RD WR ] SND : END 456 457 458# Scratch notes 459 460critical path [ READ WRITE READ ] [ READ WRITE ] [ WRITE READ ] 461--------------------> 462 > < 463Thread1 III-XXXXXX|X-SSSSSS-XX-TTTT 464Thread2 III-XXXX|XXX-TTTT 465 ^ 466 467 > XXXXXXX XX< 468 XXXX 469 470 471READ WRITE READ 472 473---> 474 475vector<instr*> {read(), write(), read()}; 476-> start() 477 478 479RECEIVE READ WRITE READ SEND 480---> 481vector<instr*> {receive(), read(), write(), read(), send()}; 482start() 483 484lock on receive() 485 486