`Feature Request: Find "orphan" files untracked by DVC, especially those in .gitignore (and fix VS Code plugin confusion)`

[frederikvand]

Hi, I'm a new DVC user, and I'm struggling to find a reliable way to make sure I haven't forgotten to track any of my data. This has been very confusing, both on the CLI and in the VS Code plugin.

The Core Problem

I follow the standard practice of adding my data/ directory to .gitignore so that Git doesn't track my large data files.

My problem is when I add new files to this data/ directory and forget to track them with DVC.

Here is my "forgotten file" scenario:

1. I have data/ listed in my .gitignore.
2. I have a file data/good_dataset.csv which is correctly tracked by DVC (it's in data.dvc).
3. Later, I download a new file, data/new_file.csv.
4. I forget to run dvc add data/new_file.csv.

Now, data/new_file.csv is an "orphan" file. It's not tracked by Git (because of .gitignore) and it's not tracked by DVC (because I forgot).

What I Tried on the CLI

I thought I could find this "orphan" file using a status command:

• git status doesn't show it (because of .gitignore). This is expected.
• dvc data status --untracked-files also doesn't show it. It seems to respect my .gitignore file, so it doesn't even look inside the data/ folder for untracked files.

This means I have no way to "audit" my repository to find files that I should have either tracked with DVC or added to .dvcignore.

This Problem is Even Worse in the VS Code Plugin

I was hoping the official DVC VS Code extension would be the solution, but it's currently unusable for this. The "DVC SCM" view is very confusing for a new user:

1. It Hides the "Orphan" Data Files: Just like the CLI, the plugin seems to respect .gitignore. My new, untracked data/new_file.csv does not show up in the plugin's file list. This means it fails to show me the exact files I'm looking for.
2. It Lists All My Code Files: At the same time, the plugin does list all of my normal code files (like train.py) that are already tracked by Git. It incorrectly suggests that I should add all my source code to DVC, which is wrong and creates a lot of noise.

This is the opposite of what I need. The plugin fails to show the data I missed, but does incorrectly show the code I've correctly tracked with Git.

My "Beginner" Expectation

As a new user, my expectation is that DVC tools should help me find data files I've forgotten to dvc add. I want a clear list of files that are:

• Not tracked by Git
• Not tracked by DVC
• Not ignored by .dvcignore

It seems the only way to do this is to write a complex custom Python script to walk the whole file system and compare git ls-files, dvc list, and dvc check-ignore, which seems very user unfriendly.

Proposed Solution

It would be incredibly helpful if there was a command or a flag to find these "orphan" files, without respecting .gitignore.

And ideally, the VS Code plugin's SCM view should be changed to reflect this. It should:

• Show untracked data files (even if in .gitignore).
• Hide code files that are already tracked by Git.

This would give me confidence that my repository is clean and I haven't forgotten to dvc add any new data. Thanks!

[skshetry]

• I have data/ listed in my .gitignore.

What is the reason to do this? DVC automatically adds datasets to .gitignore when you do dvc add, so you don't have to ignore everything.

[frederikvand]

Hi @skshetry, thanks for the quick reply. Let me explain the reason for that .gitignore rule, as it's the key to my whole issue.

What is the reason to do this?

The reason is safety in a large, multi-project repository.

This single Git repository contains ~100 different projects. The high-level .gitignore file that ignores patterns like data/ and output/ acts as a safety net. It prevents any of these 100 projects from accidentally committing large data files directly to Git.

My workflow is then to go into each project (e.g., project_A/) and run dvc add project_A/data to bring that specific folder under DVC control.

This is where the original problem begins:

1. My high-level .gitignore (the safety net) ignores all data/ folders.
2. dvc data status and the VS Code plugin respect this high-level rule.
3. As a result, they cannot "see inside" these data/ folders to find new "orphan" files that I forgot to dvc add.

So, to answer your question: No, I cannot remove that high-level gitignore. It's critical for safely managing this large repository.

The core of my feature request is that DVC's tools should be able to audit these Git-ignored directories to find files that DVC should be tracking. Does that make sense?

A key part of my confusion (especially with the VS Code plugin) is that DVC seems to be suggesting I track files that Git is already tracking.

My expectation is that DVC tools should be "Git-aware." If a file like train.py is already in Git, DVC should recognize it as "code" and automatically ignore it. It shouldn't be listed as a candidate for dvc add.

[skshetry]

Where do you track .dvc files? Do you git add --force them?

[frederikvand]

I track the .dvc files in Git.

My high-level .gitignore contains patterns like data/ and output/. When I run dvc add project_A/data, DVC creates the pointer file at project_A/data.dvc, which I then commit to git.

1. the dvc plugin lists all my code as untracked (1000s of files in different project folders)
2. dvc plugin does not show untracked folders in a project subrepo when it matches high level gitignore pattern.

[skshetry]

Here is my "forgotten file" scenario:

1. I have data/ listed in my .gitignore.
2. I have a file data/good_dataset.csv which is correctly tracked by DVC (it's in data.dvc).
3. Later, I download a new file, data/new_file.csv.
4. I forget to run dvc add data/new_file.csv.

In this scenario, what is data.dvc tracking? If it's a directory, dvc data status should show uncommitted files.

[frederikvand]

Hi @skshetry,

Thank you for helping me test this. You are absolutely correct:

In this scenario, what is data.dvc tracking? If it's a directory, dvc data status should show uncommitted files.

I've tested this, and dvc data status does successfully find new files inside an already-tracked directory, even if that directory is in .gitignore. My original assumption was wrong on that point.

---

The Real Problem: The "New Directory Blind Spot"

My core problem, which my original post didn't explain well, is the "new directory blind spot" in a multi-project repo.

Here is the refined scenario:

1. My high-level .gitignore has data/ as a safety net.
2. I have project_A/data/ which is correctly tracked by project_A/data.dvc. As you noted, dvc data status works fine here.
3. Later, I start a new project. I create project_B/data/ and add new_dataset.csv to it.
4. I forget to run dvc add project_B/data.

Because of the high-level data/ rule in .gitignore, dvc data status --untracked-files (and the VS Code plugin) never even look inside project_B/data/. The entire directory is invisible.

This is the "orphan" I can't find. There is no command in DVC that will tell me: "Hey, there's a folder at project_B/data/ that is ignored by Git but not tracked by DVC."

Proposed Solution: An "Audit" Command

This is why my feature request is for an audit command (or a flag like dvc data status --ignore-gitignore) that can find these "blind spots."

I tried writing a script to do this. Instead of posting the (buggy) code, here is the logic it uses, which is what I'm proposing DVC should do:

1. Get_All_Physical_Files: Walk the entire file system (or a sub-path) without respecting .gitignore (but ignoring .git/, .dvc/, etc.).
2. Get_Git_Files: Get the list of all files currently tracked by Git (e.g., git ls-files).
3. Get_DVC_Files: Get the list of all files currently tracked by DVC (e.g., dvc list -R --dvc-only).
4. Get_DVC_Ignored_Files: Get the list of all files explicitly ignored by .dvcignore.
5. Find Orphans: The final list of "orphan" files is:
   Get_All_Physical_Files - Get_Git_Files - Get_DVC_Files - Get_DVC_Ignored_Files

This logic completely bypasses the .gitignore "blind spot" and would correctly identify project_B/data/new_dataset.csv as a file that I need to make a decision on.

This would also solve the other major confusion with the VS Code plugin: it would see train.py in the Get_Git_Files list and correctly remove it from the "untracked by DVC" view.

Does this clarification of the "new directory blind spot" and the proposed "audit" logic make sense?