1// Copyright 2021-2023 The Khronos Group Inc.
2//
3// SPDX-License-Identifier: CC-BY-4.0
4
5= VK_EXT_calibrated_timestamps
6:toc: left
7:refpage: https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/
8:sectnums:
9
10This extension provides a way to calibrate timestamps captured from different devices in the same system and guarantees monotonicity of captured timestamps.
11
12
13== Problem Statement
14
15Vulkan has exposed timestamps since version 1.0, but does not provide a way to interpret these against wall clock time or other components in the system.
16For performance tuning and certain time-sensitive applications, getting an accurate picture of how long an operation takes and being able to calibrate these measurements against other timers is crucial.
17Two main limitations prevent this kind of measurement; lack of monotonicity, and no method of comparison.
18
19Core Vulkan 1.0 timestamps cannot be compared even across separate submits within the same run of an application, as power management events can reset the timer on many implementations.
20Comparisons within a submission are the only thing that can be relied on, and these can only really be used as a measure of relative performance compared to other workloads on the same system.
21Any solution to this problem needs to be able to guarantee a monotonic timer that only resets in the extreme circumstance that the value of the timer exceeds the size of the value that can be returned, which with 64-bit queries is, for most purposes equivalent to never happening.
22
23Once monotonicity is guaranteed, there needs to be a way to equate device timestamps to other meaningful timing systems; preferably the system timer in the host environment.
24Modern operating systems provide several ways to compare system timers to other timing systems, as well as timers at varying levels of precision.
25
26
27== Solution Space
28
29For the monotonicity guarantee, there are only really two options: either provide a monotonic clock that is undisturbed by power management events or log all power management events to the user with timestamp values stored before being reset.
30While an extension to log power management events affecting timers link:https://registry.khronos.org/OpenGL/extensions/EXT/EXT_disjoint_timer_query.txt[does exist in OpenGL ES], it only exposes whether one happened or not; to be able to generate a truly monotonic value would require more information than this.
31Additionally, a logging system would be difficult to use with timestamps consumed on the GPU; if a power management event occurs, entire frames would need to be discarded.
32To avoid this extra complication, this extension requires that timer queries are monotonic; a future extension could introduce power management event logging.
33
34As for comparing device timestamps to system time, the following options were considered:
35
36  * Add a new query type allowing the timestamp to be returned in a different domain (e.g. the host)
37  * Provide queries describing how to convert between different timer values
38  * Provide a method to capture simultaneous timestamps across different domains so they can be calibrated
39
40Returning a timestamp in a different domain via query operations is difficult as this would involve having the implementation convert the timestamps at the point they are written or copied out; which requires getting access to calibration information on the device when using vkCmdCopyQueryPoolResults, for instance.
41Queries to provide conversion information could potentially work, assuming all timers increase linearly, by providing a multiplier and offset to convert between them; however some clocks may be adjusted or otherwise drift over time, requiring implementations to update these now non-static values.
42Capturing simultaneous (or nearly simultaneous) timestamps between at least a stable host clock and the device will allow applications to calibrate timestamps at any point, which also handles adjustments or drift between clocks over time.
43
44This extension exposes a way to capture roughly simultaneous timestamps from multiple domains at once to allow them to be compared.
45
46
47== Proposal
48
49=== Monotonicity Guarantees
50
51When this extension is enabled on a device, the device timestamps returned by vkCmdWriteTimestamp must be monotonic; for any two timestamps where one happens after the other according to Vulkan synchronization guarantees, the timestamp that happens after must not return a lower timestamp value.
52The only exception to this is if the system has been running for a long enough time that the timestamp value overflows its storage, at which point it must wrap back to zero.
53For 64-bit timestamps this is practically never going to happen, but it could be observable if a 32-bit timestamp query is used.
54
55
56=== Timestamp Domains
57
58This extension provides four domains to the application; while the device domain is always available, the availability of other domains will depend on the system and the implementation, so can be queried:
59
60[source,c]
61----
62VkResult vkGetPhysicalDeviceCalibrateableTimeDomainsEXT(
63    VkPhysicalDevice                            physicalDevice,
64    uint32_t*                                   pTimeDomainCount,
65    VkTimeDomainEXT*                            pTimeDomains);
66
67typedef enum VkTimeDomainEXT {
68    VK_TIME_DOMAIN_DEVICE_EXT = 0,
69    VK_TIME_DOMAIN_CLOCK_MONOTONIC_EXT = 1,
70    VK_TIME_DOMAIN_CLOCK_MONOTONIC_RAW_EXT = 2,
71    VK_TIME_DOMAIN_QUERY_PERFORMANCE_COUNTER_EXT = 3,
72} VkTimeDomainEXT;
73----
74
75On Posix-compatible systems, `VK_TIME_DOMAIN_CLOCK_MONOTONIC_EXT` corresponds to https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/time.h.html[`CLOCK_MONOTONIC`].
76Similarly, on Linux and Linux-derived systems that support it, `VK_TIME_DOMAIN_CLOCK_MONOTONIC_RAW_EXT` corresponds to https://www.man7.org/linux/man-pages/man3/clock_gettime.3.html[`CLOCK_MONOTONIC_RAW`].
77
78For Windows systems, `VK_TIME_DOMAIN_QUERY_PERFORMANCE_COUNTER_EXT` corresponds to the time domain used by https://learn.microsoft.com/en-us/windows/win32/api/profileapi/nf-profileapi-queryperformancecounter[QueryPerformanceCounter].
79
80Finally, the `VK_TIME_DOMAIN_DEVICE_EXT` is the time domain used by the device for all link:{refpage}vkCmdWriteTimestamp.html[vkCmdWriteTimestamp] commands on any queue that supports timestamps.
81
82Implementations must always expose the device domain and at least one other domain when this extension is supported.
83
84
85=== Timestamp Calibration
86
87The following command allows applications to retrieve comparable timestamps across different domains:
88
89[source,c]
90----
91VkResult vkGetCalibratedTimestampsEXT(
92    VkDevice                                    device,
93    uint32_t                                    timestampCount,
94    const VkCalibratedTimestampInfoEXT*         pTimestampInfos,
95    uint64_t*                                   pTimestamps,
96    uint64_t*                                   pMaxDeviation);
97
98typedef struct VkCalibratedTimestampInfoEXT {
99    VkStructureType                             sType;
100    const void*                                 pNext;
101    VkTimeDomainEXT                             timeDomain;
102} VkCalibratedTimestampInfoEXT;
103----
104
105The value returned in each element of `pTimestamps` will be a raw timestamp value in the time domain specified by the corresponding element of `pTimestampInfos`.
106The returned values will be queried in their time domains as closely as possible in time, with the maximum positive deviation between the timestamps returned as a single value in `pMaxDeviation`.
107
108
109== Issues
110
111=== Should other time domain combinations be supported?
112
113As the supported set of timestamp domains is queryable, additional domains can be added over time if needed.
114
115
116=== How can an application extrapolate future device timestamp values from the calibrated timestamp value?
117
118`VkPhysicalDeviceLimits::timestampPeriod` makes it possible to calculate future device timestamps as follows:
119
120[source,c]
121----
122futureTimestamp = calibratedTimestamp + deltaNanoseconds / timestampPeriod
123----
124
125
126=== Can the host and device timestamp values drift apart over longer periods of time?
127
128Yes, especially as some time domains by definition allow for that to happen (e.g. CLOCK_MONOTONIC is subject to NTP adjustments).
129Thus it is recommended that applications re-calibrate from time to time.
130
131
132=== Should there be a query for reporting the maximum deviation of the timestamp values returned by calibrated timestamp queries?
133
134A global query seems inappropriate and difficult to enforce.
135However, it is possible to return the maximum deviation any single calibrated timestamp query can have by sampling one of the time domains twice as follows:
136
137[source,c]
138----
139timestampX = timestampX_before = SampleTimeDomain(X)
140for each time domain Y != X
141    timestampY = SampleTimeDomain(Y)
142timestampX_after = SampleTimeDomain(X)
143maxDeviation = timestampX_after - timestampX_before
144----
145
146
147=== Can the maximum deviation reported ever be zero?
148
149Unless the tick of each clock corresponding to the set of time domains coincides and all clocks can literally be sampled simultaneously, there is not really a possibility for the maximum deviation to be zero, so by convention the maximum deviation is always at least the maximum of the length of the ticks of the set of time domains calibrated and thus can never be zero.
150