aboutsummaryrefslogtreecommitdiffstats
path: root/src/util/disk_cache.c
Commit message (Collapse)AuthorAgeFilesLines
* android: Disable trying to read/write to the disk cache.Eric Anholt2020-09-221-0/+8
| | | | | | | | | | | We need the disk cache enabled in Android to get EGL_ANDROID_blob_cache's callbacks called, but we don't actually want to store anything on disk. Fixes "Failed to create //.cache for shader cache (Read-only file system)---disabling." spam on init. Reviewed-by: Tapani Pälli <tapani.palli@intel.com> Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6762>
* disk_cache: Fix filename leak on error path.Vinson Lee2020-09-171-7/+1
| | | | | | | | | | | | | | | | | Remove filename ralloc comment. filename is allocated by asprintf. Clean up disk_cache_get dead code left over from 367ac07efcc8 ("disk_cache: move cache item loading code into disk_cache_load_item() helper"). Fix defect reported by Coverity Scan. Logically dead code (DEADCODE) dead_error_line: Execution cannot reach this statement: free(filename); Signed-off-by: Vinson Lee <vlee@freedesktop.org> Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6738>
* disk_cache: move cache item loading code into disk_cache_load_item() helperTimothy Arceri2020-09-111-169/+3
| | | | | | | | | This should be helpful if someone chooses to implement cache support on windows. Also providing this greater level of abstraction makes it easier to implement alterative cache layouts in future. Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: add new OS specific helper disk_cache_evict_item()Timothy Arceri2020-09-111-12/+1
| | | | | Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: move get_cache_file() to an OS specific helperTimothy Arceri2020-09-111-25/+3
| | | | | Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: create new helper for writing cache items to diskTimothy Arceri2020-09-111-327/+3
| | | | | | | | | This pulls out the cache item writing code from cache_put() into a new helper. In this patch we also move various functions called by this code into the new disk_cache_os.c file. Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: move evict_lru_item() to an OS specific helperTimothy Arceri2020-09-111-184/+1
| | | | | Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: move munmap into an OS specific helperTimothy Arceri2020-09-111-1/+1
| | | | | Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: move index mmap into OS specific helperTimothy Arceri2020-09-111-92/+1
| | | | | | | This will make windows support easier to add in future. Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: add disk_cache_enabled() helperTimothy Arceri2020-09-111-6/+1
| | | | | | | This will make windows support easier to add in future. Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* disk_cache: move cache dir generation into OS specific helper fileTimothy Arceri2020-09-111-93/+5
| | | | | | | | | This will make windows support easier to add in future. To avoid code churn this temporarily duplicates the mkdir_if_needed() function, we will delete the duplicate in a following patch. Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
* util: stop including files from mesa/mainMarek Olšák2020-03-271-1/+0
| | | | | Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4324>
* Move compiler.h and imports.h/c from src/mesa/main into src/utilMarek Olšák2020-03-271-1/+1
| | | | | Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4324>
* glsl/tests: Fix waiting for disk_cache_put() to finish.Eric Anholt2020-03-121-0/+6
| | | | | | | | | | | | We were wasting 4s on waiting for expected-not-to-appear files to show up on every test. Using timeouts in test code is error-prone anyway, as our shared runners may be busy on other jobs. Fixes: 50989f87e62e ("util/disk_cache: use a thread queue to write to shader cache") Link: https://gitlab.freedesktop.org/mesa/mesa/issues/2505 Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4140> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4140>
* util/disk_cache: check for write() failure in the zstd pathEric Engestrom2020-02-051-1/+5
| | | | | | | | | CoverityID: 1458074 Fixes: a8d941091f72923561a6 ("util: Use ZSTD for shader cache if possible") Signed-off-by: Eric Engestrom <eric@engestrom.ch> Reviewed-by: Dylan Baker <dylan@pnwbakers.com> Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3672> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3672>
* util: Use ZSTD for shader cache if possibleDylan Baker2019-11-111-0/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows ZSTD instead of ZLIB to be used for compressing the shader cache. On a 72 core system emulating skl with a full shader-db (with i965): ZSTD: 1915.10s user 229.27s system 5150% cpu 41.632 total (cold cache) 225.40s user 10.87s system 3810% cpu 6.201 total (warm cache) 154M (235M on disk) ZLIB: 2231.33s user 194.24s system 1899% cpu 2:07.72 total (cold cache) 229.15s user 10.63s system 3906% cpu 6.139 total (warm cache) 163M (244M on disk) Tim Arceri sees (8 core ryzen and a full shader-db): ZSTD: 2505.22 user 40.50 system 3:18.73 elapsed 1280% CPU (cold cache) 418.71 user 14.93 system 0:46.53 elapsed 931% CPU (warm cache) 454.3 MB (681.7 MB on disk) ZLIB: 3069.83 user 40.02 system 4:20.13 elapsed 1195% CPU (cold cache) 425.50 user 15.17 system 0:46.80 elapsed 941% CPU (warm cache) 470.3 MB (701.4 MB on disk) Reviewed-by: Eric Engestrom <eric.engestrom@intel.com> (v1) Reviewed-by: Eric Anholt <eric@anholt.net>
* util/disk_cache: finish all queue jobs in destroy instead of killing themMarek Olšák2019-10-231-0/+1
| | | | | | | If there are queued shaders to be written to disk, wait for that. Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
* util: Workaround lack of flock on SolarisAlan Coopersmith2019-10-161-0/+10
| | | | | | | v2: Replace autoconf check for flock() with meson check Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com> Acked-by: Eric Engestrom <eric.engestrom@intel.com>
* util/disk_cache: make use of the total job size limiting featureTimothy Arceri2019-09-191-1/+1
| | | | | | | | | | | This makes use of the total job size limiting feature added in the previous patch. The idea is to avoid an excessive build up in memory use due to the use of both the UTIL_QUEUE_INIT_RESIZE_IF_FULL and UTIL_QUEUE_INIT_USE_MINIMUM_PRIORITY flags. Reviewed-by: Marek Olšák <marek.olsak@amd.com>
* util/u_queue: track job size and limit the size of queue growthTimothy Arceri2019-09-191-1/+1
| | | | | | | | | | | | | | | | | | | | | When both UTIL_QUEUE_INIT_RESIZE_IF_FULL and UTIL_QUEUE_INIT_USE_MINIMUM_PRIORITY are set, we can get into a situation where the queue never executes and grows to a huge size due to all other threads being busy. This is the case with the shader cache when attempting to compile a huge number of shaders up front. If all threads are busy compiling shaders the cache queues memory use can climb into the many GBs very fast. The use of these two flags with the shader cache is intended to allow shaders compiled at runtime to be compiled as fast as possible. To avoid huge memory use but still allow the queue to perform optimally in the run time compilation case, we now add the ability to track memory consumed by the jobs in the queue and limit it to a hardcoded 256MB which should be more than enough. Reviewed-by: Marek Olšák <marek.olsak@amd.com>
* util/disk_cache: bump thread count assigned to disk cache queueTimothy Arceri2019-09-191-3/+7
| | | | | | | | | | | | | | | Since we set the UTIL_QUEUE_INIT_USE_MINIMUM_PRIORITY flag this should have little impact on low core systems. However just about all modern CPUs currently available that run Mesa have *at least* 4 cores. For these CPUs allowing more threads can result in the queue being processed faster and avoid excessive memory use due to a backlog of cache entrys building up in the queue. This change helps avoid a huge build up of cache entrys in the queue due to using both the UTIL_QUEUE_INIT_USE_MINIMUM_PRIORITY and UTIL_QUEUE_INIT_RESIZE_IF_FULL flags. Reviewed-by: Marek Olšák <marek.olsak@amd.com>
* include: update GL headers from the registryMarek Olšák2019-06-211-0/+1
| | | | Acked-by: Ilia Mirkin <imirkin@alum.mit.edu>
* util: Heap-allocate 256K zlib bufferJory Pratt2019-06-191-1/+8
| | | | | | | | | | | | | | | | The disk cache code tries to allocate a 256 Kbyte buffer on the stack. Since musl only gives 80 Kbyte of stack space per thread, this causes a trap. See https://wiki.musl-libc.org/functional-differences-from-glibc.html#Thread-stack-size (In musl-1.1.21 the default stack size has increased to 128K) [mattst88]: Original author unknown, but I think this is small enough that it is not copyrightable. Reviewed-by: Matt Turner <mattst88@gmail.com> Reviewed-by: Eric Anholt <eric@anholt.net> Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
* util/disk_cache: close fd in the fallback pathEric Engestrom2019-03-281-4/+3
| | | | | | | | | There are multiple `goto path_fail` with an open fd, but none that go to `fail:` without going through `path_fail:` first, so let's just move the `close(fd)` there. Signed-off-by: Eric Engestrom <eric.engestrom@intel.com> Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
* util/u_queue: add UTIL_QUEUE_INIT_SET_FULL_THREAD_AFFINITYMarek Olšák2018-10-061-1/+2
| | | | | Initial version discussed with Rob Clark under a different patch name. This approach leaves his driver unaffected.
* util: rename timestamp param in disk_cache_create()Timothy Arceri2018-10-031-4/+4
| | | | | | | | Only some drivers use a timestamp here. Others use things such as build-id, or even a combination of build-ids from Mesa and LLVM. Reviewed-by: Marek Olšák <marek.olsak@amd.com>
* Shorten u_queue namesMarek Olšák2018-07-041-1/+1
| | | | | | | | There is a 15-character limit for thread names shared by the queue name and process name. Shorten the thread name to make space for the process name. Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
* disk cache: move path creation back to constructorTapani Pälli2018-02-091-86/+76
| | | | | | | | | | | This patch moves disk cache path and index creation back to the constructor which matches previous behavior. We still allow create to succeed without path so that cache can be used with callback functionality. Fixes: c95d3ed091 "disk cache: create cache even if path creation fails" Signed-off-by: Tapani Pälli <tapani.palli@intel.com> Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
* disk cache: add callback functionalityTapani Pälli2018-02-071-0/+48
| | | | | | | | | | | v2: add disk_cache_has_key, disk_cache_put_key support using blob cache (Nicolai, Jordan) v3: rename set_cb as put_cb to match existing naming (Timothy) Signed-off-by: Tapani Pälli <tapani.palli@intel.com> Reviewed-by: Jordan Justen <jordan.l.justen@intel.com> Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
* disk cache: initialize cache path and index only when usedTapani Pälli2018-02-071-49/+80
| | | | | | | | | | | | This patch makes disk_cache initialize path and index lazily so that we can utilize disk_cache without a path using callback functionality introduced by next patch. v2: unmap mmap and destroy queue only if index_mmap exists Signed-off-by: Tapani Pälli <tapani.palli@intel.com> Reviewed-by: Jordan Justen <jordan.l.justen@intel.com> Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
* mesa: include compiler.h in disk_cache.cBrian Paul2018-01-171-0/+1
| | | | | | Instead of indirect inclusion to get CPU_TO_LE32() macro. Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
* util/disk_cache: Remove unneeded free() on always null stringVadym Shovkoplias2017-12-071-1/+0
| | | | | | | | | At this point dc_job->cache_item_metadata.keys always equals NULL, so call to free() is useless Fixes: b86ecea3446 ("util/disk_cache: write cache item metadata to disk") Signed-off-by: Vadym Shovkoplias <vadym.shovkoplias@globallogic.com> Reviewed-by: Eric Engestrom <eric.engestrom@imgtec.com>
* util: Fix disk_cache index calculation on big endianMatt Turner2017-11-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The cache-test test program attempts to create a collision (using key_a and key_a_collide) by making the first two bytes identical. The idea is fine -- the shader cache wants to use the first four characters of a SHA1 hex digest as the index. The following program unsigned char array[4] = {1, 2, 3, 4}; int *ptr = (int *)array; for (int i = 0; i < 4; i++) { printf("%02x", array[i]); } printf("\n"); printf("%08x\n", *ptr); prints 01020304 04030201 on little endian, and 01020304 01020304 on big endian. On big endian platforms reading the character array back as an int (as is done in disk_cache.c) does not yield the same results as reading the byte array. To get the first four characters of the SHA1 hex digest when we mask with CACHE_INDEX_KEY_MASK, we need to byte swap the int on big endian platforms. Bugzilla: https://bugs.freedesktop.org/103668 Bugzilla: https://bugs.gentoo.org/637060 Bugzilla: https://bugs.gentoo.org/636326 Fixes: 87ab26b2ab35 ("glsl: Add initial functions to implement an on-disk cache") Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
* disk_cache: Fix issue reading GLSL metadataJordan Justen2017-10-311-1/+1
| | | | | | | | | | | | | | This would cause the read of the metadata content to fail, which would prevent the linking from being skipped. Seen on Rocket League with i965 shader cache. Fixes: b86ecea3446e "util/disk_cache: write cache item metadata to disk" Cc: Timothy Arceri <tarceri@itsqueeze.com> Signed-off-by: Jordan Justen <jordan.l.justen@intel.com> Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Reviewed-by: Jason Ekstrand <jason@jlekstrand.net> Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
* disk_cache: remove unnecessary NULL-pointer guardsNicolai Hähnle2017-10-101-4/+2
| | | | Reviewed-by: Marek Olšák <marek.olsak@amd.com>
* disk_cache: fix a memory leakNicolai Hähnle2017-10-101-0/+1
| | | | Reviewed-by: Marek Olšák <marek.olsak@amd.com>
* util/disk_cache: turn MESA_GLSL_CACHE_DISABLE into a booleanEric Engestrom2017-09-121-1/+2
| | | | | | | | | | Instead of setting based on set/unset, allow users to use boolean values. In the docs and tests, use `DISABLE=true` instead of `DISABLE=1` as it's clearer IMO. Signed-off-by: Eric Engestrom <eric.engestrom@imgtec.com> Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com> Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
* disk_cache: make the thread queue resizable and low priorityMarek Olšák2017-09-071-6/+8
| | | | | Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
* disk_cache: assert if a cache entries keys don't match mesaTimothy Arceri2017-08-281-1/+3
| | | | | | | | | In ef42423e7be9 I enabled the check for release builds however we still want to assert in debug builds in case of collisions or just general bugs with the key building/compare code. Otherwise it will just fail silently effectively disabling the cache. Reviewed-by: Eduardo Lima Mitev <elima@igalia.com>
* util/disk_cache: write cache item metadata to diskTimothy Arceri2017-08-251-1/+87
| | | | Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
* glsl: pass shader source keys to the disk cacheTimothy Arceri2017-08-251-3/+5
| | | | | | | We don't actually write them to disk here. That will happen in the following commit. Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
* disk_cache: enable limited hash collision detection in release buildsTimothy Arceri2017-08-251-16/+10
| | | | | | | It really doesn't cost us much and will stop strange crashes should the stars align. Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
* util/disk_cache: rename mesa cache dir and introduce cache versioningTimothy Arceri2017-08-251-12/+36
| | | | | | | | | | | | | | | | | | Steam is already analysing cache items, unfortunatly we did not introduce a versioning mechanism for identifying structural changes to cache entries earlier so the only way to do so is to rename the cache directory. Since we are renaming it we take the opportunity to give the directory a more meaningful name. Adding a version field to the header of cache entries will help us to avoid having to rename the directory in future. Please note this is versioning for the internal structure of the entries as defined in disk_cache.{c,h} as opposed to the structure of the data provided to the disk cache by the GLSL compiler and the various driver backends. Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
* util/u_queue: add an option to set the minimum thread priorityMarek Olšák2017-06-071-1/+1
| | | | Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
* util/disk_cache: add new driver_flags param to cache keysTimothy Arceri2017-05-231-1/+7
| | | | | | | | | This will be used for things such as adding driver specific environment variables to the key. Allowing us to set environment vars that change the shader and not have the driver ignore them if it finds existing shaders in the cache. Reviewed-by: Eduardo Lima Mitev <elima@igalia.com>
* util/disk_cache: remove percentage based max cache limitTimothy Arceri2017-04-281-6/+2
| | | | | | | | | | | | | | | | The more I think about it the more this seems like a bad idea. When we were deleting old cache dirs this wasn't so bad as it was unlikely we would ever hit the actual limit before things were cleaned up. Now that we only start cleaning up old cache items once the limit is reached the a percentage based max cache limit is more risky. For the inital release of shader cache I think its better to stick to a more conservative cache limit, at least until we have some way of cleaning up the cache more aggressively. Cc: "17.1" <mesa-stable@lists.freedesktop.org> Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
* disk_cache: use block size rather than file sizeTimothy Arceri2017-04-271-5/+11
| | | | | | | | | | | | | | | | The majority of cache files are less than 1kb this resulted in us greatly miscalculating the amount of disk space used by the cache. Using the number of blocks allocated to the file is more conservative and less likely to cause issues. This change will result in cache sizes being miscalculated further until old items added with the previous calculation have all been removed. However I don't see anyway around that, the previous patch should help limit that problem. Cc: "17.1" <mesa-stable@lists.freedesktop.org> Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
* disk_cache: reduce default cache size to 5% of filesystemTimothy Arceri2017-04-271-2/+2
| | | | | | | | | | | Modern disks are extremely large and are only going to get bigger. Usage has shown frequent Mesa upgrades can result in the cache growing very fast i.e. wasting a lot of disk space unnecessarily. 5% seems like a more reasonable default. Cc: "17.1" <mesa-stable@lists.freedesktop.org> Acked-by: Michel Dänzer <michel.daenzer@amd.com>
* util/disk_cache: don't deadlock on premature EOFGrazvydas Ignotas2017-03-251-17/+26
| | | | | | | | | | | | | | If we get EOF earlier than expected, the current read loops will deadlock. This may easily happen if the disk cache gets corrupted. Fix it by using a helper function that handles EOF. Steps to reproduce (on a build with asserts disabled): $ glxgears $ find ~/.cache/mesa/ -type f -exec truncate -s 0 '{}' \; $ glxgears # deadlock Signed-off-by: Grazvydas Ignotas <notasas@gmail.com> Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
* util/disk_cache: write cache entry keys to file headerTimothy Arceri2017-03-241-6/+44
| | | | | | | | | | | | This can be used to deal with key hash collisions from different versions (should we find that to actually happen) and to find which mesa version produced the cache entry. V2: use blob created at cache creation. v3: remove left over var from v1. Reviewed-by: Grazvydas Ignotas <notasas@gmail.com>
generated by cgit v1.2.3 (git 2.25.1) at 2025-06-16 22:16:26 +0000