Backblaze B2 S3 Remote: Consistent 403 Forbidden error on HeadObject when pushing with Master Key. #10905
Description
Bug Report
Issue name
Backblaze B2 S3 Remote: Consistent 403 Forbidden error on HeadObject when pushing with Master Key.
Description
When configuring Backblaze B2's S3-compatible API as a DVC remote and executing dvc pushusing the master application key, a 403 Forbiddenerror always occurs, causing the push to fail. The error occurs when s3fsattempts the HeadObjectoperation to verify the object's existence.
Authentication credentials (Master Key ID and Master Key) are passed via environment variables (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY), and dvc/config.local has been cleared. Since the permissions are for the master key, it should have full access to all buckets.
Reproduce
uv run dvc init
uv run dvc remote add -d b2_remote s3://dvc1102/svclust
uv run dvc remote modify b2_remote endpointurl https://s3.us-west-001.backblazeb2.com
set AWS_ACCESS_KEY_ID=XXX
set AWS_SECRET_ACCESS_KEY=YYY
set AWS_REGION=s3
uv run dvc add Data/
uv run dvc push --verbose
Expected
DVC connects to the Backblaze B2 bucket and pushes the data cache (OID) to the remote path s3://dvc1102/svclust/files/md5/... without errors.
Environment information
$ uv run dvc doctor
DVC version: 3.63.0 (pip)
Platform: Python 3.13.9 on Windows-11-10.0.26220-SP0
Subprojects:
dvc_data = 3.16.12
dvc_objects = 5.1.2
dvc_render = 1.0.2
dvc_task = 0.40.2
scmrepo = 3.5.2
Supports:
http (aiohttp = 3.13.2, aiohttp-retry = 2.9.1),
https (aiohttp = 3.13.2, aiohttp-retry = 2.9.1),
s3 (s3fs = 2025.10.0, boto3 = 1.40.61)
Config:
Global: C:\Users\gorn\AppData\Local\iterative\dvc
System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: ReFS on K:
Caches: local
Remotes: s3
Workspace directory: ReFS on K:
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\9285046a162636f50da9a2a957971b2a
Additional Information (if any):
$ uv run dvc push -v
2025-11-02 12:53:28,198 DEBUG: v3.63.0 (pip), CPython 3.13.9 on Windows-11-10.0.26220-SP0
2025-11-02 12:53:28,199 DEBUG: command: K:\Projects\My\Python\svclust.venv\Scripts\dvc push -v
Collecting |0.00 [00:00, ?entry/s]
2025-11-02 12:53:29,593 DEBUG: Preparing to transfer data from 'K:\Projects\My\Python\svclust.dvc\cache\files\md5' to 's3://dvc1102/svclust/files/md5'
2025-11-02 12:53:29,595 DEBUG: Preparing to collect status from 'dvc1102/svclust/files/md5'
2025-11-02 12:53:29,596 DEBUG: Collecting status from 'dvc1102/svclust/files/md5'
2025-11-02 12:53:29,601 DEBUG: Querying 2 oids via object_exists
Pushing
2025-11-02 12:53:33,198 ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden
Traceback (most recent call last):
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\s3fs\core.py", line 114, in _error_wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\aiobotocore\context.py", line 36, in wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\aiobotocore\client.py", line 424, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc\cli_init_.py", line 212, in main
ret = cmd.do_run()
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc\cli\command.py", line 30, in do_run
return self.run()
~~~~~~~~^^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc\commands\data_sync.py", line 65, in run
processed_files_count = self.repo.push(
targets=self.args.targets,
...<8 lines>...
glob=self.args.glob,
)
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc\repo_init_.py", line 58, in wrapper
return f(repo, *args, **kwargs)
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc\repo\push.py", line 147, in push
push_transferred, push_failed = ipush(
~~~~~^
data,
^^^^^
jobs=jobs,
^^^^^^^^^^
callback=pb.as_callback(),
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_data\index\push.py", line 78, in push
result = transfer(
cache.odb,
...<10 lines>...
callback=cb,
)
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_data\hashfile\transfer.py", line 203, in transfer
status = compare_status(
src,
...<7 lines>...
shallow=shallow,
)
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_data\hashfile\status.py", line 179, in compare_status
dest_exists, dest_missing = status(
~~~~~~^
dest,
^^^^^
...<4 lines>...
**kwargs,
^^^^^^^^^
)
^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_data\hashfile\status.py", line 139, in status
exists = hashes.intersection(
_indexed_dir_hashes(odb, index, dir_objs, name, cache_odb, jobs=jobs)
)
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_data\hashfile\status.py", line 59, in _indexed_dir_hashes
dir_exists.update(
~~~~~~~~~~~~~~~~~^
QueryingProgress(
^^^^^^^^^^^^^^^^^
...<2 lines>...
)
^
)
^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\tqdm\std.py", line 1181, in iter
for obj in iterable:
^^^^^^^^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_objects\db.py", line 366, in list_oids_exists
in_remote = self.fs.exists(paths, batch_size=jobs)
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_objects\fs\base.py", line 461, in exists
return fut.result()
~~~~~~~~~~^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.2544.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.2544.0_x64__qbz5n2kfra8p0\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\dvc_objects\executors.py", line 88, in batch_coros
result = fut.result()
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\s3fs\core.py", line 1093, in _exists
await self._info(path, bucket, key, version_id=version_id)
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\s3fs\core.py", line 1445, in _info
out = await self._call_s3(
^^^^^^^^^^^^^^^^^^^^
...<6 lines>...
)
^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\s3fs\core.py", line 371, in _call_s3
return await _error_wrapper(
^^^^^^^^^^^^^^^^^^^^^
method, kwargs=additional_kwargs, retries=self.retries
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "K:\Projects\My\Python\svclust.venv\Lib\site-packages\s3fs\core.py", line 146, in _error_wrapper
raise err
PermissionError: Forbidden
2025-11-02 12:53:33,413 DEBUG: link type reflink is not available ([Errno 129] no more link types left to try out)
2025-11-02 12:53:33,414 DEBUG: Removing 'K:\Projects\My\Python.800ffx284p9Tfoxnb-c1EA.tmp'
2025-11-02 12:53:33,416 DEBUG: Removing 'K:\Projects\My\Python.800ffx284p9Tfoxnb-c1EA.tmp'
2025-11-02 12:53:33,417 DEBUG: Removing 'K:\Projects\My\Python.800ffx284p9Tfoxnb-c1EA.tmp'
2025-11-02 12:53:33,418 DEBUG: Removing 'K:\Projects\My\Python\svclust.dvc\cache\files\md5.TsE2n1b6QqoeVPkOFflwIA.tmp'
2025-11-02 12:53:33,483 DEBUG: Version info for developers:
DVC version: 3.63.0 (pip)
Platform: Python 3.13.9 on Windows-11-10.0.26220-SP0
Subprojects:
dvc_data = 3.16.12
dvc_objects = 5.1.2
dvc_render = 1.0.2
dvc_task = 0.40.2
scmrepo = 3.5.2
Supports:
http (aiohttp = 3.13.2, aiohttp-retry = 2.9.1),
https (aiohttp = 3.13.2, aiohttp-retry = 2.9.1),
s3 (s3fs = 2025.10.0, boto3 = 1.40.61)
Config:
Global: C:\Users\gorn\AppData\Local\iterative\dvc
System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: ReFS on K:
Caches: local
Remotes: s3
Workspace directory: ReFS on K:
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\9285046a162636f50da9a2a957971b2a
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2025-11-02 12:53:33,486 DEBUG: Analytics is enabled.
2025-11-02 12:53:33,749 DEBUG: Trying to spawn ['daemon', 'analytics', 'C:\Users\gorn\AppData\Local\Temp\tmpz9e2eo5h', '-v']
2025-11-02 12:53:33,770 DEBUG: Spawned ['daemon', 'analytics', 'C:\Users\gorn\AppData\Local\Temp\tmpz9e2eo5h', '-v'] with pid 37072
exit status 255