Setup minimal round of *community facing alerts* for user home directory usage #7166
Description
This is a VEDA objective: NASA-IMPACT/veda-jupyterhub#94
Currently, admins don't know when a user's home directory is about to fill up without explicitly going and checking constantly. While ideally users should be able to clean up after themselves if their disks are full, admin intervention is still helpful as this may be a useful point to educate users about different options, as well as discover different ways in which we could serve our user population better. Currently you can do this by manually checking the grafana dashboard. This is a 'pull' operation that requires a lot of operational grunt work, that we want to avoid. We also don't know exactly how useful this would be, so it's something we want to try out and see.
We want to use this as a way for communities to explore Grafana Alerting, so they can make customized alerts that fit their own use cases. This is distinct from 2i2c's usage of alertmanager for alerts. Alertmanager alerts come to 2i2c, while these community alerts go to the community. They can then define their own processes for helping deal with these.
So a lot of our work here is defining what the communities can do, and providing tutorials / guidance on them achieving specific things. We may also want to eventually suggest workflows for the social part of various alerts, as well as learning from each other
We will use VEDA and CryoCloud communities to pilot this, as they have interested admins exploring how to do home directory work. We'll also restrict ourselves to slack as the notification channel, to simplify what we do
Definition of Done
- We have a slack channel in an appropriate slack that gets alerts when an individual user's home directories are about 80% full
- We have documentation for how a community can set this up for themselves
- This documentation has been tested out by at least one community
- We publish a blog post about this work, pointing to our documentation