This commit fixes a bug where ripgrep only treated files beginning with
a `.` as hidden. On Windows, we continue this tradition, but
additionally check whether a file has the special Windows "hidden"
attribute set. If so, we treat it as a hidden file.
In order to make this work without an additional stat call, we had to
rearrange some of the plumbing from the directory traverser.
Fixes#1154
The --ignore-file-case-insensitive flag causes all
.gitignore/.rgignore/.ignore files to have their globs matched without
regard for case. Because this introduces a potentially significant
performance regression, this is always disabled by default. Users that
need case insensitive matching can enable it on a case by case basis.
Closes#1164, Closes#1170
This also requires corresponding updates to both rand and rand_core. Doing
an update of rand without doing an update of rand_core results in
compilation errors because two distinct versions of rand_core are included
in the build, and the traits they expose are distinct and incompatible.
We also switch over to using tempfile instead of tempdir, which drops the
last remaining thing keeping rand 0.4 in the build.
Fixes#1141, Fixes#1142
This commit adds a new 'skip_stdout' option to the directory walker. When
enabled, it will skip yielding any directory entries that are believed to
correspond to stdout for the current process. This is useful for filtering
out 'results' in a command like 'grep -r foo > results' in order to avoid
an unbounded feedback mechanism.
This permits callers to sort entries by their full file path, which makes
it easy to query for various file statistics.
It would have been better to provide a comparator on DirEntry itself,
similar to how walkdir does it, but this seems to require quite a bit of
work to make the types work out, assuming we want to continue to use
walkdir's sorting support (we do).
This commit adds a 'same_file_system' option to the walk builder. For
single threaded walking, it defers to the walkdir crate, which has the
same option. The bulk of this commit implements this flag for the parallel
walker. We add one very feeble test for this.
The parallel walker is now officially a complete mess.
Closes#321
This commit fixes a bug where the first path always reported itself as
as symlink via `path_is_symlink`.
Part of this fix includes updating walkdir to 2.2.1, which also includes
a corresponding bug fix.
Fixes#984
This commit undoes a work-around for a bug in Rust's standard library
that prevented correct file type detection on Windows in OneDrive
directories. We remove the work-around because we are moving to a
latest-stable Rust version policy, which has included this fix for a while
now.
ref #705, https://github.com/rust-lang/rust/issues/46484
This also updates some code to make use of our more liberal versioning
requirement, including the use of crossbeam-channel instead of the MsQueue
from the older an unmaintained crossbeam 0.3. This does regrettably add
a sizable number of dependencies, however, compile times seem mostly
unaffected.
Closes#1019
This commit fixes an interesting bug in the `ignore` crate where it
would basically respect any `.gitignore` file anywhere (including global
gitignores in `~/.config/git/ignore`), regardless of whether we were
searching in a git repository or not. This commit rectifies that
behavior to only respect gitignore rules when in a git repo.
The key change here is to move the logic of whether to traverse parents
into the directory matcher rather than putting the onus on the directory
traverser. In particular, we now need to traverse parent directories in
more cases than we previously did, since we need to determine whether
we're in a git repository or not.
Fixes#934
When building a ignore::WalkBuilder by disabling all standard filters
and adding a custom global ignore file, the ignore file is not used. Example:
let mut walker = ignore::WalkBuilder::new(dir);
walker.standard_filters(false);
walker.add_ignore(myfile);
This makes it impossible to use the ignore crate to walk a directory
with only custom ignore files. Very similar to issue #800 (fixed in
b71a110).
PR #988
There is a small hiccup here in that a `DirEntry` can embed errors
associated with reading an ignore file, which can be accessed and logged
by consumers if desired. That error type can contain an io::Error, which
isn't cloneable. We therefore implement Clone on our library's error
type in a way that re-creates the I/O error as best as possible.
Fixes#891
This commit fixes a performance regression in Windows that resulted from
fallout from fixing #705. In particular, we introduced an additional
stat call for every single directory entry, which can be quite
disastrous for performance.
There is a corresponding companion PR that fixes the same bug in
walkdir: https://github.com/BurntSushi/walkdir/pull/96Fixes#820
This commit fixes a bug where symlinks were always being followed on
Windows, even if the user did not request it. This only impacts the
parallel iterator.
This is a regression from the fallout of fixing #705.
Fixes#824
This commit fixes a bug in the handling of custom gitignore file names.
Previously, the directory walker would check for whether there were any
ignore rules present, but this check didn't incorporate the custom gitignore
rules. At a high level, this permits custom gitignore names to be used
even if no other source of gitignore rules is used.
Fixes#800
This commit fixes a bug on Windows where directory traversals were
completely broken when attempting to scan OneDrive directories that use
the "file on demand" strategy.
The specific problem was that Rust's standard library treats OneDrive
directories as reparse points instead of directories, which causes
methods like `FileType::is_file` and `FileType::is_dir` to always return
false, even when retrieved via methods like `metadata` that purport to
follow symbolic links.
We fix this by peppering our code with checks on the underlying file
attributes exposed by Windows. We consider an entry a directory if and
only if the directory bit is set on the attributes. We are careful to
make sure that the code remains the same on non-Windows platforms.
Note that we also bump the dependency on `walkdir`, which contains a
similar fix for its traversals.
This bug is recorded upstream:
https://github.com/rust-lang/rust/issues/46484
Upstream also has a pending PR:
https://github.com/rust-lang/rust/pull/47956Fixes#705
This optimization wasn't tested too carefully, and it seems to result
in a massive amount of file handles open simultaneously. This is likely
a result of the parallel iterator, where many directories are being
traversed simultaneously.
Fixes#648
This commit fixes the symlink loop checker in the parallel directory
traverser to open fewer handles at the expense of keeping handles held
open longer.
This roughly matches the corresponding change in walkdir:
5bcc5b87eeFixes#633
The uninteresting bits of this commit involve mechanical changes for
updates to walkdir 2. The more interesting bits of this commit are the
breaking changes, although none of them should require any significant
change on users of this library. The breaking changes are as follows:
* `DirEntry::path_is_symbolic_link` has been renamed to
`DirEntry::path_is_symlink`. This matches the conventions in the
standard library, and also the corresponding name change in walkdir.
* Removed the `From<walkdir::Error> for ignore::Error` impl. This was
intended to only be used internally, but was the only thing that
made `walkdir` a public dependency of `ignore`. Therefore, we remove
it since it seems unnecessary.
* Renamed `WalkBuilder::sort_by` to `WalkBuilder::sort_by_file_name`,
and changed the type of the comparator from
Fn(&OsString, &OsString) -> cmp::Ordering + 'static
to
Fn(&OsStr, &OsStr) -> cmp::Ordering + Send + Sync + 'static
The corresponding change in `walkdir` retains the `sort_by` name, but
gives the comparator a pair of `&DirEntry` values instead of a pair
of `&OsStr` values. Ideally, `ignore` would hand off its own pair of
`&ignore::DirEntry` values, but this requires more design work. So for
now, we retain previous functionality, but leave room to make a proper
`sort_by` method.
[breaking-change]
A maximum filesize can be specified as an argument to a `WalkBuilder`.
If a file exceeds the specified size it will be ignored as part of the
resulting file/directory set.
The filesize limit never applies to directories.
When running ripgrep like this:
rg foo > output
we must be careful not to search `output` since ripgrep is actively writing
to it. Searching it can cause massive blowups where the file grows without
bound.
While this is conceptually easy to fix (check the inode of the redirection
and the inode of the file you're about to search), there are a few problems
with it.
First, inodes are a Unix thing, so we need a Windows specific solution to
this as well. To resolve this concern, I created a new crate, `same-file`,
which provides a cross platform abstraction.
Second, stat'ing every file is costly. This is not avoidable on Windows,
but on Unix, we can get the inode number directly from directory traversal.
However, this information wasn't exposed, but now it is (through both the
ignore and walkdir crates).
Fixes#286
This commit fixes two issues.
First, the iterator was executing the callback for every child of a
directory in a single thread. Therefore, if the walker was run over a
single directory, then no parallelism is used. We tweak the iterator
slightly so that we don't fall into this trap.
The second issue is a bit more subtle. In particular, we don't use the
blocking semantics of MsQueue because we *don't know when iteration
finishes*. This means that if there are a bunch of idle workers because
there is no work available to them, then they will spin and burn the
CPU. One case where this crops up is if you pipe the output of ripgrep
into `less` *and* the total number of files to search is fewer than the
number of threads ripgrep uses. We "fix" this with a very stupid
heuristic: when the queue yields no work, we sleep the thread for 1ms.
This still pegs the CPU, but not nearly as much as before.
If one really want to avoid this behavior when using ripgrep, then `-j1`
can be used to disable parallelism.
Fixes#258
When give an explicit file path on the command line like `foo` where `foo`
is a symlink, ripgrep should follow it even if `-L` isn't set. This is
consistent with the behavior of `foo/`.
Fixes#256
Previously, ignore::WalkParallel would invoke the callback for all
*explicitly* given file paths in a single thread, which effectively
meant that `rg pattern foo bar baz ...` didn't actually search foo, bar
and baz in parallel.
The code was structured that way to avoid spinning up workers if no
directory paths were given. The original intention was probably to have
a separate pool of threads responsible for searching, but ripgrep ended
up just reusing the ignore::WalkParallel workers themselves for searching,
and thereby subjected to its sub-par performance in this case.
The code has been restructured so that file paths are sent to the workers,
which brings back parallelism.
Fixes#226
Namely, passing a directory to --ignore-file caused ripgrep to allocate
memory without bound.
The issue was that I got a bit overzealous with partial error
reporting. Namely, when processing a gitignore file, we should try
to use every pattern even if some patterns are invalid globs (e.g.,
a**b). In the process, I applied the same logic to I/O errors. In this
case, it manifest by attempting to read lines from a directory, which
appears to yield Results forever, where each Result is an error of the
form "you can't read from a directory silly." Since I treated it as a
partial error, ripgrep was just spinning and accruing each error in
memory, which caused the OOM killer to kick in.
Fixes#228
This adds a new walk type in the `ignore` crate, `WalkParallel`, which
provides a way for recursively iterating over a set of paths in parallel
while respecting various ignore rules.
The API is a bit strange, as a closure producing a closure isn't
something one often sees, but it does seem to work well.
This also allowed us to simplify much of the worker logic in ripgrep
proper, where MultiWorker is now gone.
This PR introduces a new sub-crate, `ignore`, which primarily provides a
fast recursive directory iterator that respects ignore files like
gitignore and other configurable filtering rules based on globs or even
file types.
This results in a substantial source of complexity moved out of ripgrep's
core and into a reusable component that others can now (hopefully)
benefit from.
While much of the ignore code carried over from ripgrep's core, a
substantial portion of it was rewritten with the following goals in
mind:
1. Reuse matchers built from gitignore files across directory iteration.
2. Design the matcher data structure to be amenable for parallelizing
directory iteration. (Indeed, writing the parallel iterator is the
next step.)
Fixes#9, #44, #45
