aboutsummaryrefslogtreecommitdiffstats
path: root/finder
Commit message (Collapse)AuthorAgeFilesLines
* Reimplement ioutil.ReadDir with a version that avoids calling lstatColin Cross2017-12-224-1/+547
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ioutil.ReadDir returns []os.FileInfo, which contains information on each entry in the directory that is only available by calling os.Lstat on the entry. Finder only the name and type (regular, directory or symlink) of the files, which on Linux kernels >= 2.6.4 is available in the return values of syscall.Getdents. Replace ioutil.ReadDir with a call that uses syscall.Getdents directly and collects the type information from the result. Testing with: rm -f /tmp/db && strace -fc finder -names Android.mk,Android.bp,Blueprints,CleanSpec.mk,TEST_MAPPING -exclude-dirs .git,.repo -prune-files .out-dir,.find-ignore -db /tmp/db . Before: 7.01 52.688304 63 833398 1 lstat 1.90 14.246644 68 210523 getdents64 1.25 9.370471 90 104286 1 openat After: 3.48 12.201385 117 104286 1 openat 3.06 10.729138 51 210523 getdents64 1.70 5.951892 57 104283 1 lstat Pros: Avoids 729115 calls to lstat. Cons: Requires copying ~200 lines of finicky buffer parsing code. Puts all getdents calls (and possibly fallback lstat calls) onto a non-blocking file descriptor, which will cause it to block a thread and not just a goroutine. Only works on Linux and Darwin. Bug: 70897635 Test: m checkbuild Change-Id: Iab9f82c38c8675d0b73b4e90540bb9e4d2ee52c1
* Add a DirEntryInfo interface that is a subset of os.FileInfoColin Cross2017-12-222-6/+29
| | | | | | | | | | | | | | | | ioutil.ReadDir returns []os.FileInfo, which contains information on each entry in the directory that is only available by calling os.Lstat on the entry. Finder only the name and type (regular, directory or symlink) of the files, which on Linux kernels >= 2.6.4 is available in the return values of syscall.Getdents. In preparation for using syscall.Getdents, switch filesystem.ReadDir to return an interface that only contains the information that will be available from syscall.Getdents. Bug: 70897635 Test: m checkbuild Change-Id: Id2749d709a0f7b5a61abedde68549d4bf208a568
* Move android/soong/fs to android/soong/finder/fsColin Cross2017-12-228-5/+1074
| | | | | | | | The fs package is specific to finder, move it inside finder. Bug: 70897635 Test: m checkbuild Change-Id: Ie705f064a832141702a8e87fd59ed75c01018504
* Fix the Finder's ability to ignore permission errorsJeff Gaston2017-08-231-1/+1
| | | | | | | | | | It turns out the os was returning a different permissions error than os.ErrPermission. Fortunately we can just use os.IsPermission(). Bug: 62455338 Test: mkdir not-permitted && chmod 000 not-permitted && m nothing Change-Id: Ieb1653b9687ffdaae83184dcdcd4166ddb6a4613
* Add some tests for singlethreaded usage of the Finder.Jeff Gaston2017-08-222-58/+100
| | | | | | | | | | | Also change the existing tests to use exactly 2 threads. Also update some tests to also same number of threads as will be used by the finder. Bug: 62455338 Test: m -j # which runs unit tests Change-Id: I2b9d39680ecddd6d28c951df982ac51c077d2777
* Fix hanging of finder_testJeff Gaston2017-08-221-0/+3
| | | | | | | | | | | | | | | | | | | Previously there could be an infinite loop in the Finder's db export when running tests on a system with few cores. The Finder divides the cache-serialization work amongst cores. On systems with enough cores, each path in TestFileSystemRoot would get its own core and the infinite loop in the prefix calculation logic would be avoided. On systems with fewer cores, the computation of the maximum common path could run forever. This shouldn't have affected the general usage of the Finder in builds other than in its unit tests, because the builds don't ask to include the filesystem root as a path to scan. This should only have affected finder_test.go which tested the filesystem root anyway. Bug: 62455338 Test: m -j # which runs unit tests Change-Id: I7a1467fb32e4538fb96833791180bfbfad5a3911
* Run the Finder and make its results available to KatiJeff Gaston2017-08-162-13/+38
| | | | | | | | | | The Finder runs roughly 200ms faster than findleaves.py in aosp, and runs roughly 400ms faster in internal master. Bug: 64363847 Test: m -j Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
* Fail the Finder in case of unexpected fs errorJeff Gaston2017-08-153-36/+201
| | | | | | | | | Permissions errors are ignored Errors on pruned dirs are also ignored Bug: 62455338 Test: m -j blueprint_tools # which runs unit tests Change-Id: I8ba85fdd0295deb7dc374a851212e7c850e76b75
* Revert "Revert "Cacheable, multithreaded finder.""Jeff Gaston2017-08-105-0/+3187
| | | | | | | | | Bug: 62455338 Test: m -j This reverts commit d1abeb9d982b11fdf4047176d213acc8197c375f. Change-Id: I9f73031636157511b5f1c6ce8a205e9bc91669ff
* Revert "Cacheable, multithreaded finder."Jeff Gaston2017-08-105-3187/+0
| | | | | | | | This reverts commit b6d161bf16031e06fc8d532e35a53c73ad20f84f. Reason for revert: New Build Breakage: aosp-master/sdk_mac @ 4260825 Change-Id: I8bda8c50c5e5c9f84621d11a4c15b168833bcd21
* Cacheable, multithreaded finder.Jeff Gaston2017-08-085-0/+3187
It can find every Android.bp in internal master in about 2.5 sec the first time and 0.3 sec subsequent times Bug: 62455338 Test: m -j blueprint_tools # which runs the unit tests Test: m -j blueprint_tools && \ out/soong/host/linux-x86/bin/finder \ -v --db /tmp/mydb \ --names Android.mk \ --prune-files .android-out-dir \ --exclude-dirs .git,.repo \ . \ >/tmp/finder-log 2>&1 Change-Id: I5ab2650459a1dae0d5d076faf411ec2d053c743d
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-22 14:45:07 +0000