1// Copyright 2021-2023 The Khronos Group Inc. 2// 3// SPDX-License-Identifier: CC-BY-4.0 4 5= VK_EXT_calibrated_timestamps 6:toc: left 7:refpage: https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/ 8:sectnums: 9 10This extension provides a way to calibrate timestamps captured from different devices in the same system and guarantees monotonicity of captured timestamps. 11 12 13== Problem Statement 14 15Vulkan has exposed timestamps since version 1.0, but does not provide a way to interpret these against wall clock time or other components in the system. 16For performance tuning and certain time-sensitive applications, getting an accurate picture of how long an operation takes and being able to calibrate these measurements against other timers is crucial. 17Two main limitations prevent this kind of measurement; lack of monotonicity, and no method of comparison. 18 19Core Vulkan 1.0 timestamps cannot be compared even across separate submits within the same run of an application, as power management events can reset the timer on many implementations. 20Comparisons within a submission are the only thing that can be relied on, and these can only really be used as a measure of relative performance compared to other workloads on the same system. 21Any solution to this problem needs to be able to guarantee a monotonic timer that only resets in the extreme circumstance that the value of the timer exceeds the size of the value that can be returned, which with 64-bit queries is, for most purposes equivalent to never happening. 22 23Once monotonicity is guaranteed, there needs to be a way to equate device timestamps to other meaningful timing systems; preferably the system timer in the host environment. 24Modern operating systems provide several ways to compare system timers to other timing systems, as well as timers at varying levels of precision. 25 26 27== Solution Space 28 29For the monotonicity guarantee, there are only really two options: either provide a monotonic clock that is undisturbed by power management events or log all power management events to the user with timestamp values stored before being reset. 30While an extension to log power management events affecting timers link:https://registry.khronos.org/OpenGL/extensions/EXT/EXT_disjoint_timer_query.txt[does exist in OpenGL ES], it only exposes whether one happened or not; to be able to generate a truly monotonic value would require more information than this. 31Additionally, a logging system would be difficult to use with timestamps consumed on the GPU; if a power management event occurs, entire frames would need to be discarded. 32To avoid this extra complication, this extension requires that timer queries are monotonic; a future extension could introduce power management event logging. 33 34As for comparing device timestamps to system time, the following options were considered: 35 36 * Add a new query type allowing the timestamp to be returned in a different domain (e.g. the host) 37 * Provide queries describing how to convert between different timer values 38 * Provide a method to capture simultaneous timestamps across different domains so they can be calibrated 39 40Returning a timestamp in a different domain via query operations is difficult as this would involve having the implementation convert the timestamps at the point they are written or copied out; which requires getting access to calibration information on the device when using vkCmdCopyQueryPoolResults, for instance. 41Queries to provide conversion information could potentially work, assuming all timers increase linearly, by providing a multiplier and offset to convert between them; however some clocks may be adjusted or otherwise drift over time, requiring implementations to update these now non-static values. 42Capturing simultaneous (or nearly simultaneous) timestamps between at least a stable host clock and the device will allow applications to calibrate timestamps at any point, which also handles adjustments or drift between clocks over time. 43 44This extension exposes a way to capture roughly simultaneous timestamps from multiple domains at once to allow them to be compared. 45 46 47== Proposal 48 49=== Monotonicity Guarantees 50 51When this extension is enabled on a device, the device timestamps returned by vkCmdWriteTimestamp must be monotonic; for any two timestamps where one happens after the other according to Vulkan synchronization guarantees, the timestamp that happens after must not return a lower timestamp value. 52The only exception to this is if the system has been running for a long enough time that the timestamp value overflows its storage, at which point it must wrap back to zero. 53For 64-bit timestamps this is practically never going to happen, but it could be observable if a 32-bit timestamp query is used. 54 55 56=== Timestamp Domains 57 58This extension provides four domains to the application; while the device domain is always available, the availability of other domains will depend on the system and the implementation, so can be queried: 59 60[source,c] 61---- 62VkResult vkGetPhysicalDeviceCalibrateableTimeDomainsEXT( 63 VkPhysicalDevice physicalDevice, 64 uint32_t* pTimeDomainCount, 65 VkTimeDomainEXT* pTimeDomains); 66 67typedef enum VkTimeDomainEXT { 68 VK_TIME_DOMAIN_DEVICE_EXT = 0, 69 VK_TIME_DOMAIN_CLOCK_MONOTONIC_EXT = 1, 70 VK_TIME_DOMAIN_CLOCK_MONOTONIC_RAW_EXT = 2, 71 VK_TIME_DOMAIN_QUERY_PERFORMANCE_COUNTER_EXT = 3, 72} VkTimeDomainEXT; 73---- 74 75On Posix-compatible systems, `VK_TIME_DOMAIN_CLOCK_MONOTONIC_EXT` corresponds to https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/time.h.html[`CLOCK_MONOTONIC`]. 76Similarly, on Linux and Linux-derived systems that support it, `VK_TIME_DOMAIN_CLOCK_MONOTONIC_RAW_EXT` corresponds to https://www.man7.org/linux/man-pages/man3/clock_gettime.3.html[`CLOCK_MONOTONIC_RAW`]. 77 78For Windows systems, `VK_TIME_DOMAIN_QUERY_PERFORMANCE_COUNTER_EXT` corresponds to the time domain used by https://learn.microsoft.com/en-us/windows/win32/api/profileapi/nf-profileapi-queryperformancecounter[QueryPerformanceCounter]. 79 80Finally, the `VK_TIME_DOMAIN_DEVICE_EXT` is the time domain used by the device for all link:{refpage}vkCmdWriteTimestamp.html[vkCmdWriteTimestamp] commands on any queue that supports timestamps. 81 82Implementations must always expose the device domain and at least one other domain when this extension is supported. 83 84 85=== Timestamp Calibration 86 87The following command allows applications to retrieve comparable timestamps across different domains: 88 89[source,c] 90---- 91VkResult vkGetCalibratedTimestampsEXT( 92 VkDevice device, 93 uint32_t timestampCount, 94 const VkCalibratedTimestampInfoEXT* pTimestampInfos, 95 uint64_t* pTimestamps, 96 uint64_t* pMaxDeviation); 97 98typedef struct VkCalibratedTimestampInfoEXT { 99 VkStructureType sType; 100 const void* pNext; 101 VkTimeDomainEXT timeDomain; 102} VkCalibratedTimestampInfoEXT; 103---- 104 105The value returned in each element of `pTimestamps` will be a raw timestamp value in the time domain specified by the corresponding element of `pTimestampInfos`. 106The returned values will be queried in their time domains as closely as possible in time, with the maximum positive deviation between the timestamps returned as a single value in `pMaxDeviation`. 107 108 109== Issues 110 111=== Should other time domain combinations be supported? 112 113As the supported set of timestamp domains is queryable, additional domains can be added over time if needed. 114 115 116=== How can an application extrapolate future device timestamp values from the calibrated timestamp value? 117 118`VkPhysicalDeviceLimits::timestampPeriod` makes it possible to calculate future device timestamps as follows: 119 120[source,c] 121---- 122futureTimestamp = calibratedTimestamp + deltaNanoseconds / timestampPeriod 123---- 124 125 126=== Can the host and device timestamp values drift apart over longer periods of time? 127 128Yes, especially as some time domains by definition allow for that to happen (e.g. CLOCK_MONOTONIC is subject to NTP adjustments). 129Thus it is recommended that applications re-calibrate from time to time. 130 131 132=== Should there be a query for reporting the maximum deviation of the timestamp values returned by calibrated timestamp queries? 133 134A global query seems inappropriate and difficult to enforce. 135However, it is possible to return the maximum deviation any single calibrated timestamp query can have by sampling one of the time domains twice as follows: 136 137[source,c] 138---- 139timestampX = timestampX_before = SampleTimeDomain(X) 140for each time domain Y != X 141 timestampY = SampleTimeDomain(Y) 142timestampX_after = SampleTimeDomain(X) 143maxDeviation = timestampX_after - timestampX_before 144---- 145 146 147=== Can the maximum deviation reported ever be zero? 148 149Unless the tick of each clock corresponding to the set of time domains coincides and all clocks can literally be sampled simultaneously, there is not really a possibility for the maximum deviation to be zero, so by convention the maximum deviation is always at least the maximum of the length of the ticks of the set of time domains calibrated and thus can never be zero. 150