Skip to content

Commit 9e61427

Browse files
feat: add GCS support as file provider (#2013) (#2435)
* feat(file/gcs): add GCS support with Go 1.25 and fix requested changes * added gcs filestore and some testcase * Added GCS FIlestore * added testcase for Stat * format code using goimports and fix lint issues via golangci-lint * fix lint issue * refactor: update docs, logging, and contribution guidelines based on feedback * test: add missing coverage for GCS * docs(gcs): add EndPoint field in config example * refactor(tests): remove if-else by splitting into separate test functions * feat(file/gcs): add GCS support with Go 1.25 and fix requested changes * Added GCS FIlestore * added testcase for Stat * format code using goimports and fix lint issues via golangci-lint * fix lint issue * refactor: update docs, logging, and contribution guidelines based on feedback * docs(gcs): add EndPoint field in config example * refactor(tests): remove if-else by splitting into separate test functions and fix linting issue * feat(file/gcs): add GCS support with Go 1.25 and fix requested changes * added gcs filestore and some testcase * Added GCS FIlestore * added testcase for Stat * format code using goimports and fix lint issues via golangci-lint * fix lint issue * test: add missing coverage for GCS * docs(gcs): add EndPoint field in config example * refactor(tests): remove if-else by splitting into separate test functions * feat(file/gcs): add GCS support with Go 1.25 and fix requested changes * Added GCS FIlestore * added testcase for Stat * format code using goimports and fix lint issues via golangci-lint * fix lint issue * refactor: update docs, logging, and contribution guidelines based on feedback * docs(gcs): add EndPoint field in config example * refactor(tests): remove if-else by splitting into separate test functions and fix linting issue * fix: handle nil metrics in Connect * extract out logger and metrics and use them for diff file providers * use LogFileoperation method to log and record metrics for gcs implementation and removing separate logger for gcs * rename common Log method and resolve linters * update seek writeAt and readAt methods * resolve review comments * refactor gcs implementation and abstracted common file and directories operations to file directory * add tests for gcs package files * add tests for common file system methods in file package * resolve linter issues * further simplify implementation by abstracting out all common file operations * resolve linters * fix failing tests and linters * fix failing test * fix errors in s3 and sftp * remove errors from ftp and s3 package * add tests * add tests * update implementation to remove common fields from gcs and other providers to shift to common file system implementation * restore New method to prevent breaking change * fix linters * update docs --------- Co-authored-by: Suryakant <dassuryakantadas@gmail.com>
1 parent 5860444 commit 9e61427

30 files changed

+5791
-1992
lines changed

CONTRIBUTING.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ docker run -d --name db -p 8091-8096:8091-8096 -p 11210-11211:11210-11211 couchb
102102
docker login container-registry.oracle.com
103103
docker pull container-registry.oracle.com/database/free:latest
104104
docker run -d --name oracle-free -p 1521:1521 -e ORACLE_PWD=YourPasswordHere container-registry.oracle.com/database/free:latest
105+
docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest
105106
```
106107
107108
> [!NOTE]
@@ -144,4 +145,4 @@ docker run -d --name oracle-free -p 1521:1521 -e ORACLE_PWD=YourPasswordHere con
144145
- Use trailing white space or the <br> HTML tag at the end of the line.
145146
- Use "`" sign to add single line code and "```" to add multi-line code block.
146147
- Use relative references to images (in `public` folder as mentioned above.)
147-
* The [gofr.dev documentation](https://gofr.dev/docs) site is updated upon push to `/docs` path in the repo. Verify your changes are live after next GoFr version.
148+
* The [gofr.dev documentation](https://gofr.dev/docs) site is updated upon push to `/docs` path in the repo. Verify your changes are live after next GoFr version.

docs/advanced-guide/handling-file/page.md

Lines changed: 107 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,15 @@ GoFr simplifies the complexity of working with different file stores by offering
66

77
By default, local file-store is initialized and user can access it from the context.
88

9-
GoFr also supports FTP/SFTP file-store. Developers can also connect and use their AWS S3 bucket as a file-store. The file-store can be initialized as follows:
9+
GoFr also supports FTP/SFTP file-store. Developers can also connect and use their cloud storage bucket as a file-store. Following cloud storage options are currently supported:
10+
11+
- **AWS S3**
12+
- **Google Cloud Storage (GCS)**
13+
14+
The file-store can be initialized as follows:
1015

1116
### FTP file-store
17+
1218
```go
1319
package main
1420

@@ -34,6 +40,7 @@ func main() {
3440
```
3541

3642
### SFTP file-store
43+
3744
```go
3845
package main
3946

@@ -60,8 +67,7 @@ func main() {
6067
### AWS S3 Bucket as File-Store
6168

6269
To run S3 File-Store locally we can use localstack,
63-
``docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack``
64-
70+
`docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack`
6571

6672
```go
6773
package main
@@ -90,17 +96,83 @@ func main() {
9096
app.Run()
9197
}
9298
```
93-
> Note: The current implementation supports handling only one bucket at a time,
94-
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.
99+
100+
> Note: The current implementation supports handling only one bucket at a time,
101+
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.
102+
103+
### Google Cloud Storage (GCS) Bucket as File-Store
104+
105+
To run GCS File-Store locally we can use fake-gcs-server:
106+
`docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest`
107+
108+
```go
109+
package main
110+
111+
import (
112+
"gofr.dev/pkg/gofr"
113+
114+
"gofr.dev/pkg/gofr/datasource/file/gcs"
115+
)
116+
117+
func main() {
118+
app := gofr.New()
119+
120+
// Option 1: Using JSON credentials with local emulator
121+
fs, err := gcs.New(&gcs.Config{
122+
EndPoint: "http://localhost:4566",
123+
BucketName: "my-bucket",
124+
CredentialsJSON: readFile("gcs-credentials.json"),
125+
ProjectID: "my-project-id",
126+
})
127+
128+
if err != nil {
129+
app.Logger().Fatalf("Failed to initialize GCS: %v", err)
130+
131+
}
132+
133+
app.AddFileStore(fs)
134+
135+
// Option 2: Using default credentials (GOOGLE_APPLICATION_CREDENTIALS)
136+
// fs, err := gcs.New(&gcs.Config{
137+
// BucketName: "my-bucket",
138+
// ProjectID: "my-project-id",
139+
// }))
140+
141+
// Option 3: Direct connection to real GCS (no EndPoint)
142+
// fs, err := gcs.New(&gcs.Config{
143+
// BucketName: "my-bucket",
144+
// CredentialsJSON: readFile("prod-creds.json"),
145+
// ProjectID: "my-project-id",
146+
// }))
147+
148+
app.Run()
149+
}
150+
151+
// Helper function to read credentials file
152+
func readFile(filename string) []byte {
153+
data, err := os.ReadFile(filename)
154+
if err != nil {
155+
log.Fatalf("Failed to read credentials file: %v", err)
156+
}
157+
return data
158+
}
159+
160+
```
161+
162+
> **Note:** When connecting to the actual GCS service, authentication can be provided via CredentialsJSON or the GOOGLE_APPLICATION_CREDENTIALS environment variable.
163+
> When using fake-gcs-server, authentication is not required.
164+
> Currently supports one bucket per file-store instance.
95165
96166
### Creating Directory
97167

98168
To create a single directory
169+
99170
```go
100171
err := ctx.File.Mkdir("my_dir",os.ModePerm)
101172
```
102173

103174
To create subdirectories as well
175+
104176
```go
105177
err := ctx.File.MkdirAll("my_dir/sub_dir", os.ModePerm)
106178
```
@@ -114,27 +186,32 @@ currentDir, err := ctx.File.Getwd()
114186
### Change current Directory
115187

116188
To switch to parent directory
189+
117190
```go
118191
currentDir, err := ctx.File.Chdir("..")
119192
```
120193

121194
To switch to another directory in same parent directory
195+
122196
```go
123197
currentDir, err := ctx.File.Chdir("../my_dir2")
124198
```
125199

126200
To switch to a subfolder of the current directory
201+
127202
```go
128203
currentDir, err := ctx.File.Chdir("sub_dir")
129204
```
205+
130206
> Note: This method attempts to change the directory, but S3's flat structure and fixed bucket
131-
> make this operation inapplicable.
207+
> make this operation inapplicable. Similarly, GCS uses a flat structure where directories are simulated through object prefixes.
132208
133209
### Read a Directory
134210

135-
The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.
211+
The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.
136212

137213
If an error occurs during the read operation, ReadDir returns the successfully read entries up to the point of the error along with the error itself. Passing "." as the directory argument returns the entries for the current directory.
214+
138215
```go
139216
entries, err := ctx.File.ReadDir("../testdir")
140217

@@ -143,12 +220,13 @@ for _, entry := range entries {
143220

144221
if entry.IsDir() {
145222
entryType = "Dir"
146-
}
223+
}
147224

148225
fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
149226
}
150227
```
151-
> Note: In S3, directories are represented as prefixes of file keys. This method retrieves file
228+
229+
> Note: In S3 and GCS, directories are represented as prefixes of file keys/object names. This method retrieves file
152230
> entries only from the immediate level within the specified directory.
153231
154232
### Creating and Save a File with Content
@@ -163,14 +241,15 @@ _, _ = file.Write([]byte("Hello World!"))
163241
```
164242

165243
### Reading file as CSV/JSON/TEXT
244+
166245
GoFr support reading CSV/JSON/TEXT files line by line.
167246

168247
```go
169248
reader, err := file.ReadAll()
170249

171250
for reader.Next() {
172251
var b string
173-
252+
174253
// For reading CSV/TEXT files user need to pass pointer to string to SCAN.
175254
// In case of JSON user should pass structs with JSON tags as defined in encoding/json.
176255
err = reader.Scan(&b)
@@ -179,10 +258,12 @@ for reader.Next() {
179258
}
180259
```
181260

182-
183261
### Opening and Reading Content from a File
262+
184263
To open a file with default settings, use the `Open` command, which provides read and seek permissions only. For write permissions, use `OpenFile` with the appropriate file modes.
264+
185265
> Note: In FTP, file permissions are not differentiated; both `Open` and `OpenFile` allow all file operations regardless of specified permissions.
266+
186267
```go
187268
csvFile, _ := ctx.File.Open("my_file.csv")
188269

@@ -205,6 +286,7 @@ if err != nil {
205286
### Getting Information of the file/directory
206287

207288
Stat retrieves details of a file or directory, including its name, size, last modified time, and type (such as whether it is a file or folder)
289+
208290
```go
209291
file, _ := ctx.File.Stat("my_file.text")
210292
entryType := "File"
@@ -215,10 +297,12 @@ if entry.IsDir() {
215297

216298
fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
217299
```
218-
>Note: In S3:
300+
301+
> Note: In S3 and GCS:
302+
>
219303
> - Names without a file extension are treated as directories by default.
220-
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed.
221-
>
304+
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed (S3 specific behavior).
305+
>
222306
> For directories, the method calculates the total size of all contained objects and returns the most recent modification time. For files, it directly returns the file's size and last modified time.
223307
224308
### Rename/Move a File
@@ -234,18 +318,23 @@ err := ctx.File.Rename("old_name.text", "new_name.text")
234318
### Deleting Files
235319

236320
`Remove` deletes a single file
237-
> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method.
321+
322+
> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method. GCS supports deletion of both files and empty directories.
323+
238324
```go
239325
err := ctx.File.Remove("my_dir")
240326
```
241327

242328
The `RemoveAll` command deletes all subdirectories as well. If you delete the current working directory, such as "../currentDir", the working directory will be reset to its parent directory.
243-
> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided.
329+
330+
> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided for S3.
331+
> GCS handles both files and directories.
332+
244333
```go
245334
err := ctx.File.RemoveAll("my_dir/my_text")
246335
```
247336

248-
> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 uses
249-
> a flat file structure, all methods require a full path relative to the S3 bucket.
337+
> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 and GCS use
338+
> a flat file structure, all methods require a full path relative to the bucket.
250339
251340
> Errors have been skipped in the example to focus on the core logic, it is recommended to handle all the errors.

go.work

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ use (
1111
./pkg/gofr/datasource/elasticsearch
1212
./pkg/gofr/datasource/file/ftp
1313
./pkg/gofr/datasource/file/s3
14+
./pkg/gofr/datasource/file/gcs
1415
./pkg/gofr/datasource/file/sftp
1516
./pkg/gofr/datasource/influxdb
1617
./pkg/gofr/datasource/kv-store/badger

0 commit comments

Comments
 (0)