[ANE-2672] Add --x-vendetta flag (#1607)

* Rough draft

* Build source unit

* Return full ficus analysis result

* Propogate full FicusAnalysisResults

* Extract ficus result source unit

* Use proper metadata format

* Insignificant change

* Update to accept specific strategies

* Fix merge mistake

* Improve FicusSpec

* Lint

* Add vendetta doc

* Update changelog

* Update ficus cmd

* Add reference to vendetta docs

* Address PR comments

* Document more accurate timing

* Fix typo

* Use correct snippet scan strategy name

* Only show snippet scan result logs if snippet scan is requested

* Pattern match acc

* Reword error

* Move FicusSpec to unit tests and use Test.Effect

* Add doc about doing an inital local scan

* Simplify spec

Author: nficca

Changelog.md

@@ -1,5 +1,8 @@
 # FOSSA CLI Changelog
 
+## 3.14.0
+- Adds `--x-vendetta` flag for vendored dependency identification ([#1607](https://github.com/fossas/fossa-cli/pull/1607))
+
 ## 3.13.1
 - Add a summary of the snippet scan when the `--x-snippet-scan` flag is used ([#1613](https://github.com/fossas/fossa-cli/pull/1613))
 - Update snippet scanning documentation ([#1615](https://github.com/fossas/fossa-cli/pull/1615))

docs/features/vendetta.md

@@ -0,0 +1,106 @@
+
+# Vendetta
+
+Vendetta is the name of FOSSA's vendored dependency identification feature.
+
+Vendetta hashes files in your first party source code, compares them against
+FOSSA's knowledge base, and matches them to common open source components before
+finally feeding those matches to a special algorithm that deduces a holistic set
+of vendored open source dependencies present in your project.
+
+Vendetta can be run as part of `fossa analyze`. To enable it, add the
+`--x-vendetta` flag when you run `fossa analyze`:
+
+```sh
+fossa analyze --x-vendetta
+```
+
+## How Vendetta Works
+
+When `--x-vendetta` is enabled, the CLI:
+
+1. **Hashes Files**: Creates MD5 hashes of the contents of all relevant files.
+2. **Filters Content**: By default, skips directories like `.git/`, and hidden
+    directories. This includes, from `.fossa.yml`,
+    `vendoredDependencies.licenseScanPathFilters.exclude`, documented further
+    below.
+5. **Uploads Hashes**: Sends only the hashes to FOSSA's servers.
+6. **Receives Matches**: Gets back information about any matching open source
+   components.
+7. **Infers Dependencies**: Feeds the matches to an algorithm that heuristically
+   identifies the vendored dependencies in your project.
+
+## Data Sent to FOSSA
+
+Vendetta sends _only_ the MD5 hashes of your file contents to FOSSA. The raw
+contents are never sent to FOSSA.
+
+## Data Retention
+
+The MD5 hashes are stored permanently in FOSSA.
+
+## Directory Filtering
+
+By default, Vendetta excludes common non-production directories and follows
+`.gitignore` patterns:
+
+- Hidden directories.
+- Globs as directed by `.gitignore` files.
+
+#### Custom Exclude Filtering
+
+You can customize which files and directories are excluded from Vendetta by
+configuring exclude filters in your `.fossa.yml` file. Note that Vendetta scans
+currently only support exclude patterns, not `only` patterns.
+
+For example:
+```yaml
+version: 3
+vendoredDependencies:
+  licenseScanPathFilters:
+    exclude:
+      - "**/test/**"
+      - "**/tests/**"
+      - "**/spec/**"
+      - "**/node_modules/**"
+      - "**/dist/**"
+      - "**/build/**"
+      - "**/*.test.js"
+      - "**/*.spec.ts"
+```
+
+**Important Notes:**
+
+- Vendetta scanning only use the `exclude` filters from `licenseScanPathFilters`
+  — `only` filters are ignored for this use-case.
+- Path filters use standard glob patterns (e.g., `**/*` for recursive matching,
+  `*` for single-directory matching).
+- The configuration goes in the
+  `vendoredDependencies.licenseScanPathFilters.exclude` section.
+- These exclude patterns are passed directly to the Ficus scanning engine as
+  `--exclude` arguments.
+- Default exclusions (hidden files, `.gitignore` patterns) are applied in
+  addition to custom excludes.
+
+## A note on scan times
+
+The first time you run Vendetta on a codebase, it may take a long time to scan.
+For example, scanning [Linux](https://github.com/torvalds/linux) for the first
+time may take upwards of 60 minutes. This is because most of the files in your
+codebase will have never been checked against FOSSA's knowledge base for open
+source components, which can take time.
+
+Once you scan the first time however, FOSSA will cache the open source component
+matches for each MD5 hash Vendetta provides. This means that subsequent scans of
+the same project will be drastically faster. For example, scanning the same
+revision of Linux twice in a row should result in the second scan taking only
+1-2 minutes.
+
+The time it takes to scan newer versions of your codebase will depend on how
+many files in the new version have not been previously scanned. A file has been
+previously scanned if the exact same file has ever been scanned by Vendetta.
+FOSSA recommends scanning your codebase on a regular basis to keep scan times
+low. Additionally, if you intend on running Vendetta as part of your CI
+pipeline, it might be best to do a manual run first on a local machine. That
+way, future automated scans of your project will be able to benefit from the
+initial caching done in the first scan.

docs/references/subcommands/analyze.md

@@ -158,6 +158,25 @@ Snippet Scanning must also be enabled for your organization, and is only availab
 
 For more detail about how Snippet Scanning works, how to use file filtering during Snippet Scanning, what information is sent to FOSSA's servers and a description of the Snippet Scan Summary, see [the Snippet Scanning feature documentation](../../features/snippet-scanning.md).
 
+### Vendored Dependency Scanning with Vendetta
+
+Vendetta is a feature that identifies the paths of potential open source code
+dependencies vendored in your project by comparing file hashes against FOSSA's
+knowledge base. This feature helps find dependencies that are included in your
+project directly as source.
+
+#### Enabling Vendetta
+
+| Name                | Description                                                                                                                                                                           |
+|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `--x-vendetta`      | Enable vendored dependency scanning during analysis. This experimental feature hashes your source files and checks them against FOSSA's open source component database.               |
+
+#### More detail
+
+For more detail about how Vendetta works, how to use file filtering during
+scanning, or what information is sent to FOSSA's servers, see
+[the Vendetta feature documentation](../../features/vendetta.md).
+
 ### Experimental Options
 
 _Important: For support and other general information, refer to the [experimental options overview](../experimental/README.md) before using experimental options._

integration-test/Analysis/FicusSpec.hs

@@ -657,6 +657,7 @@ test-suite unit-tests
     Erlang.Rebar3TreeSpec
     Extra.ListSpec
     Extra.TextSpec
+    Ficus.FicusSpec
     Fortran.FpmTomlSpec
     Fossa.API.TypesSpec
     Go.GlideLockSpec
@@ -758,7 +759,6 @@ test-suite integration-tests
     Analysis.CocoapodsSpec
     Analysis.ElixirSpec
     Analysis.ErlangSpec
-    Analysis.FicusSpec
     Analysis.FixtureExpectationUtils
     Analysis.FixtureUtils
     Analysis.GoSpec

spectrometer.cabal

@@ -51,6 +51,7 @@ import App.Fossa.Config.Analyze (
 import App.Fossa.Config.Analyze qualified as Config
 import App.Fossa.Config.Common (DestinationMeta (..), destinationApiOpts, destinationMetadata)
 import App.Fossa.Ficus.Analyze (analyzeWithFicus)
+import App.Fossa.Ficus.Types (FicusAnalysisResults (vendoredDependencyScanResults), FicusStrategy (FicusStrategySnippetScan, FicusStrategyVendetta), FicusVendoredDependencyScanResults (FicusVendoredDependencyScanResults))
 import App.Fossa.FirstPartyScan (runFirstPartyScan)
 import App.Fossa.Lernie.Analyze (analyzeWithLernie)
 import App.Fossa.Lernie.Types (LernieResults (..))
@@ -103,12 +104,12 @@ import Data.Flag (Flag, fromFlag)
 import Data.Foldable (traverse_)
 import Data.Functor (($>))
 import Data.List.NonEmpty qualified as NE
-import Data.Maybe (fromMaybe, isJust, mapMaybe)
+import Data.Maybe (catMaybes, fromMaybe, isJust, mapMaybe, maybeToList)
 import Data.String.Conversion (decodeUtf8, toText)
 import Data.Text.Extra (showT)
 import Data.Traversable (for)
 import Diag.Diagnostic as DI
-import Diag.Result (Result (Success), resultToMaybe)
+import Diag.Result (resultToMaybe)
 import Discovery.Archive qualified as Archive
 import Discovery.Filters (AllFilters, MavenScopeFilters, applyFilters, filterIsVSIOnly, ignoredPaths, isDefaultNonProductionPath)
 import Discovery.Projects (withDiscoveredProjects)
@@ -302,6 +303,7 @@ analyze cfg = Diag.context "fossa-analyze" $ do
       allowedTactics = Config.allowedTacticTypes cfg
       withoutDefaultFilters = Config.withoutDefaultFilters cfg
       enableSnippetScan = Config.xSnippetScan cfg
+      enableVendetta = Config.xVendetta cfg
 
   manualSrcUnits <-
     Diag.errorBoundaryIO . diagToDebug $
@@ -340,27 +342,27 @@ analyze cfg = Diag.context "fossa-analyze" $ do
         if (fromFlag BinaryDiscovery $ Config.binaryDiscoveryEnabled $ Config.vsiOptions cfg)
           then analyzeDiscoverBinaries basedir filters
           else pure Nothing
+  let ficusStrategies =
+        catMaybes
+          [ if enableSnippetScan then Just FicusStrategySnippetScan else Nothing
+          , if enableVendetta then Just FicusStrategyVendetta else Nothing
+          ]
   maybeFicusResults <-
     Diag.errorBoundaryIO . diagToDebug $
-      if not enableSnippetScan
+      if null ficusStrategies || filterIsVSIOnly filters
         then do
-          logInfo "Skipping ficus snippet scanning (--x-snippet-scan not set)"
           pure Nothing
         else
-          if filterIsVSIOnly filters
-            then do
-              logInfo "Running in VSI only mode, skipping snippet-scan"
-              pure Nothing
-            else
-              Diag.context "snippet-scanning"
-                . runStickyLogger SevInfo
-                $ analyzeWithFicus
-                  basedir
-                  maybeApiOpts
-                  revision
-                  (Config.licenseScanPathFilters vendoredDepsOptions)
-                  (orgSnippetScanSourceCodeRetentionDays =<< orgInfo)
-                  (Config.debugDir cfg)
+          Diag.context "ficus-scanning"
+            . runStickyLogger SevInfo
+            $ analyzeWithFicus
+              basedir
+              maybeApiOpts
+              revision
+              ficusStrategies
+              (Config.licenseScanPathFilters vendoredDepsOptions)
+              (orgSnippetScanSourceCodeRetentionDays =<< orgInfo)
+              (Config.debugDir cfg)
   let ficusResults = join $ resultToMaybe maybeFicusResults
 
   maybeLernieResults <-
@@ -378,13 +380,22 @@ analyze cfg = Diag.context "fossa-analyze" $ do
       vsiResults' :: [SourceUnit]
       vsiResults' = fromMaybe [] $ join (resultToMaybe vsiResults)
 
+      ficusResults' :: [SourceUnit]
+      ficusResults' =
+        maybeToList $
+          ficusResults
+            >>= vendoredDependencyScanResults
+            >>= \(FicusVendoredDependencyScanResults maybeSrcUnit) -> maybeSrcUnit
+
       additionalSourceUnits :: [SourceUnit]
-      additionalSourceUnits = vsiResults' <> mapMaybe (join . resultToMaybe) [manualSrcUnits, binarySearchResults, dynamicLinkedResults]
+      additionalSourceUnits = vsiResults' <> ficusResults' <> mapMaybe (join . resultToMaybe) [manualSrcUnits, binarySearchResults, dynamicLinkedResults]
   traverse_ (Diag.flushLogs SevError SevDebug) [manualSrcUnits, binarySearchResults, dynamicLinkedResults]
   -- Flush logs using the original Result from VSI.
   traverse_ (Diag.flushLogs SevError SevDebug) [vsiResults]
   -- Flush logs from lernie
   traverse_ (Diag.flushLogs SevError SevDebug) [maybeLernieResults]
+  -- Flush logs from ficus
+  traverse_ (Diag.flushLogs SevError SevDebug) [maybeFicusResults]
 
   maybeFirstPartyScanResults <-
     Diag.errorBoundaryIO . diagToDebug $
@@ -450,7 +461,7 @@ analyze cfg = Diag.context "fossa-analyze" $ do
           $ analyzeForReachability projectScans
   let reachabilityUnits = onlyFoundUnits reachabilityUnitsResult
 
-  let analysisResult = AnalysisScanResult projectScans vsiResults binarySearchResults (Success [] Nothing) manualSrcUnits dynamicLinkedResults maybeLernieResults reachabilityUnitsResult
+  let analysisResult = AnalysisScanResult projectScans vsiResults binarySearchResults maybeFicusResults manualSrcUnits dynamicLinkedResults maybeLernieResults reachabilityUnitsResult
       isDebugMode = isJust (Config.debugDir cfg)
   renderScanSummary isDebugMode maybeEndpointAppVersion analysisResult cfg
 

src/App/Fossa/Analyze.hs

@@ -12,7 +12,7 @@ module App.Fossa.Analyze.Types (
 
 import App.Fossa.Analyze.Project (ProjectResult)
 import App.Fossa.Config.Analyze (ExperimentalAnalyzeConfig)
-import App.Fossa.Ficus.Types (FicusSnippetScanResults)
+import App.Fossa.Ficus.Types (FicusAnalysisResults)
 import App.Fossa.Lernie.Types (LernieResults)
 import App.Fossa.Reachability.Types (SourceUnitReachability (..))
 import App.Types (Mode)
@@ -81,7 +81,7 @@ data AnalysisScanResult = AnalysisScanResult
   { analyzersScanResult :: [DiscoveredProjectScan]
   , vsiScanResult :: Result (Maybe [SourceUnit])
   , binaryDepsScanResult :: Result (Maybe SourceUnit)
-  , ficusResult :: Result (Maybe FicusSnippetScanResults)
+  , ficusResult :: Result (Maybe FicusAnalysisResults)
   , fossaDepsScanResult :: Result (Maybe SourceUnit)
   , dynamicLinkingResult :: Result (Maybe SourceUnit)
   , lernieResult :: Result (Maybe LernieResults)