Skip to content
Open
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
511923d
adding new dependency and class
tcnichol Mar 28, 2024
f763cb4
starting a user migration
tcnichol Mar 28, 2024
84d43bc
adding script for migrate metadata definitions
tcnichol Mar 28, 2024
e82e023
adding option of temporary password (for use with users who are being…
tcnichol Mar 28, 2024
dcdbea1
reverting changes
tcnichol Mar 28, 2024
79e56b4
temporary - for users that are migrated
tcnichol Mar 28, 2024
33f1cab
creates users
tcnichol Apr 23, 2024
93509e7
does not actually create the dataset
tcnichol Apr 25, 2024
aba43a3
using beanie might not work here due to async issues
tcnichol Apr 25, 2024
727685f
using nest asyncio to fix problem with even loop closing
tcnichol Apr 25, 2024
d4e9526
entry from right table for file bytes
tcnichol Apr 26, 2024
8c7d6bd
dependencies for migration, adding file entry and indexing files
tcnichol May 20, 2024
2ba7df5
wrong api key
tcnichol May 20, 2024
f624784
api key works for user,
tcnichol May 21, 2024
adc6c63
file upload works, sloppy needs fixed
tcnichol May 21, 2024
171b2b3
we are only getting partial files, not sure how to fix yet
tcnichol May 21, 2024
1f6a2ed
files upload right now
tcnichol May 26, 2024
2bf100c
delete file
tcnichol May 26, 2024
4eccae1
adding folders
tcnichol May 27, 2024
c01ce96
folder hierarchy - this should work
tcnichol May 30, 2024
c2ab289
works for now, folders need to be fixed
tcnichol May 31, 2024
2d7cdce
previous is no longer selectable when moving to my datasets. we start…
tcnichol Jun 1, 2024
65599bb
Merge branch 'main' into 978-migrate-users
tcnichol Jun 1, 2024
5be97bf
new folder hierarchy method works
tcnichol Jun 1, 2024
f479284
need a license for datasets
tcnichol Jun 2, 2024
cb4b835
method for getting all the folders
tcnichol Jun 2, 2024
e885a2a
need to remove logging, but folders are created and files uploaded to…
tcnichol Jun 3, 2024
b4db0f4
should work now
tcnichol Jun 3, 2024
72e0845
fixing typos
tcnichol Jun 3, 2024
a6cd25a
formatting
tcnichol Jun 3, 2024
54747ca
sample .env
tcnichol Jun 3, 2024
55ff21a
formatting
tcnichol Jun 3, 2024
e216405
Merge branch 'main' into 978-migrate-users
tcnichol Jul 13, 2024
851503e
refactoring migrate_user
tcnichol Jul 13, 2024
074154e
formatting
tcnichol Jul 15, 2024
909b8f6
Merge branch 'main' into 978-migrate-users
tcnichol Aug 5, 2024
b309023
Merge branch 'release/v2.0-beta-3' into 978-migrate-users
tcnichol Aug 6, 2024
58ada46
Merge remote-tracking branch 'origin/main' into 978-migrate-users
tcnichol Aug 14, 2024
62d5b02
rename
tcnichol Aug 22, 2024
8772325
Merge branch 'release/v2.0-beta-3' into 978-migrate-users
longshuicy Aug 22, 2024
ec56aa4
migrate metadata definition; modify migration script (#1184)
longshuicy Aug 26, 2024
8050460
fix clowder user api key creation issue
longshuicy Aug 27, 2024
c936760
fix user api key if user already exists
tcnichol Aug 27, 2024
5547715
Merge remote-tracking branch 'origin/978-migrate-users' into 978-migr…
tcnichol Aug 27, 2024
86c8493
switch headers and files
tcnichol Aug 27, 2024
2a52c7c
Migrate metadata (#1192)
longshuicy Aug 29, 2024
4e68f21
1187 migrate collections and spaces as v2 metadata. (#1190)
tcnichol Sep 3, 2024
4d5281a
Merge branch 'main' into 978-migrate-users
tcnichol Aug 16, 2025
70ea8af
removed - already processed
tcnichol Aug 16, 2025
92f4f2c
new readme, using tomli and a config file to limit or set options for…
tcnichol Aug 17, 2025
80d41b0
adding example for config.toml
tcnichol Aug 17, 2025
bc67fc7
exclude users - users we do not migrate (or their files)
tcnichol Aug 17, 2025
d86a756
adding example env
tcnichol Aug 17, 2025
a453068
proper path to .env file
tcnichol Aug 17, 2025
c64be06
excluding items
tcnichol Aug 17, 2025
c7e218c
check to see if we should migrate a dataset
tcnichol Aug 17, 2025
c4d3cdb
using the excluded space ids
tcnichol Aug 17, 2025
ca80dbd
fix - if license absent in v1 assign a default license for v2
tcnichol Aug 17, 2025
f98edda
fix type in license
tcnichol Aug 17, 2025
5a84349
adding TODO items
tcnichol Aug 17, 2025
1932362
working on this, need a better way to check what collections a datase…
tcnichol Aug 17, 2025
21629c7
sloppy hack - keeping for commit since it's got the descendant collec…
tcnichol Aug 18, 2025
b600cc9
new classes (to be consolidated later)
tcnichol Aug 18, 2025
734c9f1
this seems to be working
tcnichol Aug 18, 2025
87324ec
instructions for now (rough draft)
tcnichol Aug 18, 2025
55a54bb
metadata posts
tcnichol Aug 18, 2025
dbe3016
log metadata def - errors are when there are duplicates
tcnichol Aug 19, 2025
73609d2
get top level collections
tcnichol Aug 21, 2025
093bfb0
method stubs for new way of processing
tcnichol Aug 22, 2025
8477cba
more method stubs
tcnichol Aug 22, 2025
f2a48a7
new methods
tcnichol Aug 23, 2025
9b9cbed
organize based on the type of parent, need to work out this logic
tcnichol Aug 23, 2025
b34eb84
this should work up to the datasets in a root collection, need testin…
tcnichol Aug 23, 2025
a5dd8ea
remove unused methods
tcnichol Aug 24, 2025
f534160
too many folders created
tcnichol Aug 24, 2025
a00e600
progress for dataset folders and files
tcnichol Aug 25, 2025
2fb33c4
the folder hierarchy part is not working for v2
tcnichol Aug 26, 2025
10f2026
new route should return all folders in a dataset
tcnichol Aug 27, 2025
19fb46b
new route for returning all folders in a dataset
tcnichol Aug 28, 2025
4640c19
folder hierarchy is now properly created
tcnichol Aug 28, 2025
ff8c62c
new method upload to folder - we find the folder, may need to depreca…
tcnichol Aug 28, 2025
1a5e2b7
new method should work for uploading files to either a dataset or a f…
tcnichol Aug 29, 2025
04fc4b2
partial fix?
tcnichol Sep 1, 2025
c76e75e
messy, but works, can post the data
tcnichol Sep 2, 2025
c45d8f0
worked for one collection with datasets and folders
tcnichol Sep 3, 2025
9ca8850
other collection also works now
tcnichol Sep 3, 2025
e2be15b
get list of datasets in collections, we need this for migration
tcnichol Sep 3, 2025
c94dd6d
this should work for datasets now as well
tcnichol Sep 3, 2025
88e2534
single dataset works, except metadata
tcnichol Sep 3, 2025
7e139cb
removing unused methods
tcnichol Sep 3, 2025
5b2562d
adding todo with dataset metadata to a folder
tcnichol Sep 3, 2025
0a8e764
migrate now adds a metadata file to a folder
tcnichol Sep 8, 2025
19b0f02
running pipenv run black
tcnichol Sep 8, 2025
0fa7465
running pipenv run black for app
tcnichol Sep 8, 2025
80ae7cf
fix typo that got in
tcnichol Sep 8, 2025
a034ca5
ran codegen?
tcnichol Sep 8, 2025
1f3e5ed
ran codegen?
tcnichol Sep 8, 2025
34c868b
more readme instructions
tcnichol Sep 9, 2025
27a1355
fixing lint
tcnichol Nov 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,17 @@ deployments/kubernetes/charts/clowder2/*clowder2-software-dev.yaml
*secret*.yaml

# Environments
**/*.env
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
*/.env
scripts/migration/.env


# Test folder contect except for test source code and metadata document.
backend/app/tests/*
Expand Down
5 changes: 4 additions & 1 deletion backend/app/keycloak_auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,9 @@ async def get_current_user_id(identity: Json = Depends(get_token)) -> str:
return keycloak_id


async def create_user(email: str, password: str, firstName: str, lastName: str):
async def create_user(
email: str, password: str, firstName: str, lastName: str, temporary: bool = False
):
"""Create a user in Keycloak."""
keycloak_admin = KeycloakAdmin(
server_url=settings.auth_server_url,
Expand All @@ -427,6 +429,7 @@ async def create_user(email: str, password: str, firstName: str, lastName: str):
{
"value": password,
"type": "password",
"temporary": temporary,
}
],
},
Expand Down
Loading