summaryrefslogtreecommitdiff
path: root/finder
AgeCommit message (Collapse)Author
2024-01-18Fix data race in finder_test.go Colin Cross
Wait for newFinder to finish writing its dump before ticking the clock. Test: go test -race ./... Change-Id: Ib7527ebfb4e1c989deefdb0779a4f9c0928ae15d
2022-08-17Reformat build/soong for go 1.19 Colin Cross
Test: none Change-Id: I132368f0fcbdb5ea088b5b84dbe4ccfdd9e94cad
2022-05-04Add a test for correctness of C++ compilation. Lukacs T. Berki
This required the following: - Adding Platform_base_sdk_extension_version to default soong.variables - Teaching the symlink tree creation code to understand symlinks - Making finder.go follow symlinks when requested Adding yet another knob is unfortunate, but I can't allow that unconditionally because the Android code base contains a number of symlinks giving rise to infinite directory trees because they point back to their parent and this seemed preferable to adding complicated logic like "follow symlink but if only its fully resolved version does not point under the source tree". I could be convinced about the latter, though. Test: Presubmits. Change-Id: I453f6b7e5334771f5832c700db00f9d24ed1d82f
2022-02-07Close more files. Liz Kammer
Test: m nothing Change-Id: I4914337feb4ee2499c270b8504a097a2fd597ab3
2021-02-06Add LOCAL_LICENSE_KINDS to build/soong Bob Badour
Added SPDX-license-identifier-Apache-2.0 to: Android.bp android/Android.bp android/soongconfig/Android.bp androidmk/Android.bp apex/Android.bp bazel/Android.bp bp2build/Android.bp bpf/Android.bp bpfix/Android.bp cc/Android.bp cc/config/Android.bp cc/libbuildversion/Android.bp cc/libbuildversion/tests/Android.bp cc/ndk_api_coverage_parser/Android.bp cc/ndkstubgen/Android.bp cc/symbolfile/Android.bp cmd/dep_fixer/Android.bp cmd/diff_target_files/Android.bp cmd/extract_apks/Android.bp cmd/extract_jar_packages/Android.bp cmd/extract_linker/Android.bp cmd/fileslist/Android.bp cmd/host_bionic_inject/Android.bp cmd/javac_wrapper/Android.bp cmd/merge_zips/Android.bp cmd/multiproduct_kati/Android.bp cmd/path_interposer/Android.bp cmd/pom2bp/Android.bp cmd/pom2mk/Android.bp cmd/sbox/Android.bp cmd/soong_build/Android.bp cmd/soong_env/Android.bp cmd/soong_ui/Android.bp cmd/zip2zip/Android.bp cmd/zipsync/Android.bp cuj/Android.bp dexpreopt/Android.bp dexpreopt/dexpreopt_gen/Android.bp env/Android.bp etc/Android.bp filesystem/Android.bp finder/Android.bp finder/cmd/Android.bp genrule/Android.bp jar/Android.bp java/Android.bp java/config/Android.bp kernel/Android.bp linkerconfig/Android.bp linkerconfig/proto/Android.bp makedeps/Android.bp partner/Android.bp phony/Android.bp python/Android.bp python/tests/Android.bp remoteexec/Android.bp rust/Android.bp rust/config/Android.bp scripts/Android.bp sdk/Android.bp sh/Android.bp shared/Android.bp symbol_inject/Android.bp symbol_inject/cmd/Android.bp sysprop/Android.bp tradefed/Android.bp ui/build/Android.bp ui/logger/Android.bp ui/metrics/Android.bp ui/metrics/proc/Android.bp ui/status/Android.bp ui/terminal/Android.bp ui/tracer/Android.bp xml/Android.bp zip/Android.bp zip/cmd/Android.bp Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to: finder/fs/Android.bp third_party/zip/Android.bp Bug: 68860345 Bug: 151177513 Bug: 151953481 Test: m all Exempt-From-Owner-Approval: janitorial work Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2020-10-13find bazel-related files and add them to bazel.list and ninja deps Chris Parsons
This retriggers soong_build whenever a new bzl, WORKSPACE, or BUILD.bazel file is changed or added. Test: Manually verified on bionic/libc genrules with manual changes to related BUILD/bzl/WORKSPACE files -- these all retrigger builds. Test: Updated finder_test.go Change-Id: I634384f88781a6b6db32f5d6bf9c07e179e14c39
2020-10-09Revert "Make lots of tests run in parallel" Colin Cross
This reverts commit 323dc60712491c71ccdc5363c42df61f0a192487. Reason for revert: Possible cause of test instability Bug: 170513220 Test: soong tests Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-06Make lots of tests run in parallel Colin Cross
Putting t.Parallel() in each test makes them run in parallel. Additional t.Parallel() could be added to each subtest, although that requires making a local copy of the loop variable for table driven tests. Test: m checkbuild Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-07-06Dist module_paths/files.db Colin Cross
Help debugging finder issues by disting module_paths/files.db for later analysis. Bug: 157656545 Test: treehugger Change-Id: Idfa4cffe7ef724b399a2667efd487f9c3c76bb6b
2020-07-01Merge changes Ie33d2e05,Ie2b4509b,I5ac3a1f5 Colin Cross
* changes: Fix finder on symlinks pointing to directories Add Stat to finder/fs Move finder_test filesystem helper functions to fs/test.go
2020-06-29Fix finder on symlinks pointing to directories Colin Cross
When finder encountered a symlink pointing to a directory it attempted to ignore it, but becuase Lstat returns the a *os.FileInfo for the symlink the child.IsDir() check always returned false. Call Stat on the symlink to get the *os.FileInfo for the target of the symlink. Bug: 157656545 Test: finder_test.go Change-Id: Ie33d2e05d5c435b48e83eaeadf7b3c9816398404
2020-06-29Add Stat to finder/fs Colin Cross
Add a Stat method to finder/fs that will be used by finder to read the mode of the target of a symlink. Bug: 157656545 Test: fs_test.go Change-Id: Ie2b4509b7d11857d9a1685de4477088b91d43c63
2020-06-29Move finder_test filesystem helper functions to fs/test.go Colin Cross
Move the filesystem helper functions used by finder_test.go into the fs package so they can also be used by fs tests. Bug: 157656545 Test: m checkbuild Change-Id: I5ac3a1f502d8e2cb7c15dd70ca85a46d20f939a6
2020-06-29Fix bpfmt issues and add bpfmt to preupload checks Thiébaud Weksteen
Test: m nothing Change-Id: I10526a33685335bdd26cc7692815133379d4e0f6
2019-06-20Fix data races in finder_test.go Colin Cross
Fix data races by incrementing clock before starting the finder. Fixes: ================== WARNING: DATA RACE Read at 0x00c0003be1c0 by goroutine 99: android/soong/finder/fs.(*MockFs).WriteFile() build/soong/finder/fs/fs.go:159 +0x869 android/soong/finder.(*Finder).dumpDb() build/soong/finder/finder.go:1163 +0x296 android/soong/finder.(*Finder).goDumpDb.func1() build/soong/finder/finder.go:337 +0x3c Previous write at 0x00c0003be1c0 by goroutine 43: android/soong/finder.TestDirectoryNotPermitted() build/soong/finder/fs/fs.go:155 +0x49a testing.tRunner() prebuilts/go/linux-x86/src/testing/testing.go:865 +0x163 Goroutine 99 (running) created at: android/soong/finder.(*Finder).goDumpDb() build/soong/finder/finder.go:336 +0x8d android/soong/finder.(*Finder).loadFromFilesystem() build/soong/finder/finder.go:380 +0x12b android/soong/finder.newImpl() build/soong/finder/finder.go:197 +0x38d android/soong/finder.newFinderAndErr() build/soong/finder/finder_test.go:58 +0x245 android/soong/finder.newFinderWithNumThreads() build/soong/finder/finder_test.go:42 +0xb3 android/soong/finder.TestDirectoryNotPermitted() build/soong/finder/finder_test.go:38 +0x386 testing.tRunner() prebuilts/go/linux-x86/src/testing/testing.go:865 +0x163 Goroutine 43 (running) created at: testing.(*T).Run() prebuilts/go/linux-x86/src/testing/testing.go:916 +0x65a testing.runTests.func1() prebuilts/go/linux-x86/src/testing/testing.go:1157 +0xa8 testing.tRunner() prebuilts/go/linux-x86/src/testing/testing.go:865 +0x163 testing.runTests() prebuilts/go/linux-x86/src/testing/testing.go:1155 +0x523 testing.(*M).Run() prebuilts/go/linux-x86/src/testing/testing.go:1072 +0x2eb main.main() out/soong/.bootstrap/soong-finder/test/test.go:157 +0x1dc ================== Test: run finder_test.go with race detector enabled Change-Id: Ia77090dabaf8b14a80ac9ce938fef75f1d382574
2018-07-22Fix `go vet` issues Dan Willemsen
Test: go vet ./... Change-Id: Ifb936ccc5e2b5a2c3fcbbbcb54f680e2973ea1b3
2017-12-22Reimplement ioutil.ReadDir with a version that avoids calling lstat Colin Cross
ioutil.ReadDir returns []os.FileInfo, which contains information on each entry in the directory that is only available by calling os.Lstat on the entry. Finder only the name and type (regular, directory or symlink) of the files, which on Linux kernels >= 2.6.4 is available in the return values of syscall.Getdents. Replace ioutil.ReadDir with a call that uses syscall.Getdents directly and collects the type information from the result. Testing with: rm -f /tmp/db && strace -fc finder -names Android.mk,Android.bp,Blueprints,CleanSpec.mk,TEST_MAPPING -exclude-dirs .git,.repo -prune-files .out-dir,.find-ignore -db /tmp/db . Before: 7.01 52.688304 63 833398 1 lstat 1.90 14.246644 68 210523 getdents64 1.25 9.370471 90 104286 1 openat After: 3.48 12.201385 117 104286 1 openat 3.06 10.729138 51 210523 getdents64 1.70 5.951892 57 104283 1 lstat Pros: Avoids 729115 calls to lstat. Cons: Requires copying ~200 lines of finicky buffer parsing code. Puts all getdents calls (and possibly fallback lstat calls) onto a non-blocking file descriptor, which will cause it to block a thread and not just a goroutine. Only works on Linux and Darwin. Bug: 70897635 Test: m checkbuild Change-Id: Iab9f82c38c8675d0b73b4e90540bb9e4d2ee52c1
2017-12-22Add a DirEntryInfo interface that is a subset of os.FileInfo Colin Cross
ioutil.ReadDir returns []os.FileInfo, which contains information on each entry in the directory that is only available by calling os.Lstat on the entry. Finder only the name and type (regular, directory or symlink) of the files, which on Linux kernels >= 2.6.4 is available in the return values of syscall.Getdents. In preparation for using syscall.Getdents, switch filesystem.ReadDir to return an interface that only contains the information that will be available from syscall.Getdents. Bug: 70897635 Test: m checkbuild Change-Id: Id2749d709a0f7b5a61abedde68549d4bf208a568
2017-12-22Move android/soong/fs to android/soong/finder/fs Colin Cross
The fs package is specific to finder, move it inside finder. Bug: 70897635 Test: m checkbuild Change-Id: Ie705f064a832141702a8e87fd59ed75c01018504
2017-08-23Fix the Finder's ability to ignore permission errors Jeff Gaston
It turns out the os was returning a different permissions error than os.ErrPermission. Fortunately we can just use os.IsPermission(). Bug: 62455338 Test: mkdir not-permitted && chmod 000 not-permitted && m nothing Change-Id: Ieb1653b9687ffdaae83184dcdcd4166ddb6a4613
2017-08-22Add some tests for singlethreaded usage of the Finder. Jeff Gaston
Also change the existing tests to use exactly 2 threads. Also update some tests to also same number of threads as will be used by the finder. Bug: 62455338 Test: m -j # which runs unit tests Change-Id: I2b9d39680ecddd6d28c951df982ac51c077d2777
2017-08-22Fix hanging of finder_test Jeff Gaston
Previously there could be an infinite loop in the Finder's db export when running tests on a system with few cores. The Finder divides the cache-serialization work amongst cores. On systems with enough cores, each path in TestFileSystemRoot would get its own core and the infinite loop in the prefix calculation logic would be avoided. On systems with fewer cores, the computation of the maximum common path could run forever. This shouldn't have affected the general usage of the Finder in builds other than in its unit tests, because the builds don't ask to include the filesystem root as a path to scan. This should only have affected finder_test.go which tested the filesystem root anyway. Bug: 62455338 Test: m -j # which runs unit tests Change-Id: I7a1467fb32e4538fb96833791180bfbfad5a3911
2017-08-16Run the Finder and make its results available to Kati Jeff Gaston
The Finder runs roughly 200ms faster than findleaves.py in aosp, and runs roughly 400ms faster in internal master. Bug: 64363847 Test: m -j Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
2017-08-15Fail the Finder in case of unexpected fs error Jeff Gaston
Permissions errors are ignored Errors on pruned dirs are also ignored Bug: 62455338 Test: m -j blueprint_tools # which runs unit tests Change-Id: I8ba85fdd0295deb7dc374a851212e7c850e76b75
2017-08-10Revert "Revert "Cacheable, multithreaded finder."" Jeff Gaston
Bug: 62455338 Test: m -j This reverts commit d1abeb9d982b11fdf4047176d213acc8197c375f. Change-Id: I9f73031636157511b5f1c6ce8a205e9bc91669ff
2017-08-10Revert "Cacheable, multithreaded finder." Jeff Gaston
This reverts commit b6d161bf16031e06fc8d532e35a53c73ad20f84f. Reason for revert: New Build Breakage: aosp-master/sdk_mac @ 4260825 Change-Id: I8bda8c50c5e5c9f84621d11a4c15b168833bcd21
2017-08-08Cacheable, multithreaded finder. Jeff Gaston
It can find every Android.bp in internal master in about 2.5 sec the first time and 0.3 sec subsequent times Bug: 62455338 Test: m -j blueprint_tools # which runs the unit tests Test: m -j blueprint_tools && \ out/soong/host/linux-x86/bin/finder \ -v --db /tmp/mydb \ --names Android.mk \ --prune-files .android-out-dir \ --exclude-dirs .git,.repo \ . \ >/tmp/finder-log 2>&1 Change-Id: I5ab2650459a1dae0d5d076faf411ec2d053c743d
generated by cgit v1.2.3-59-g8ed1b (git 2.39.0) at 2025-10-19 05:41:18 +0000