You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat: add GCS support as file provider (#2013) (#2435)
* feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
* added gcs filestore and some testcase
* Added GCS FIlestore
* added testcase for Stat
* format code using goimports and fix lint issues via golangci-lint
* fix lint issue
* refactor: update docs, logging, and contribution guidelines based on feedback
* test: add missing coverage for GCS
* docs(gcs): add EndPoint field in config example
* refactor(tests): remove if-else by splitting into separate test functions
* feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
* Added GCS FIlestore
* added testcase for Stat
* format code using goimports and fix lint issues via golangci-lint
* fix lint issue
* refactor: update docs, logging, and contribution guidelines based on feedback
* docs(gcs): add EndPoint field in config example
* refactor(tests): remove if-else by splitting into separate test functions and fix linting issue
* feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
* added gcs filestore and some testcase
* Added GCS FIlestore
* added testcase for Stat
* format code using goimports and fix lint issues via golangci-lint
* fix lint issue
* test: add missing coverage for GCS
* docs(gcs): add EndPoint field in config example
* refactor(tests): remove if-else by splitting into separate test functions
* feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
* Added GCS FIlestore
* added testcase for Stat
* format code using goimports and fix lint issues via golangci-lint
* fix lint issue
* refactor: update docs, logging, and contribution guidelines based on feedback
* docs(gcs): add EndPoint field in config example
* refactor(tests): remove if-else by splitting into separate test functions and fix linting issue
* fix: handle nil metrics in Connect
* extract out logger and metrics and use them for diff file providers
* use LogFileoperation method to log and record metrics for gcs implementation and removing separate logger for gcs
* rename common Log method and resolve linters
* update seek writeAt and readAt methods
* resolve review comments
* refactor gcs implementation and abstracted common file and directories operations to file directory
* add tests for gcs package files
* add tests for common file system methods in file package
* resolve linter issues
* further simplify implementation by abstracting out all common file operations
* resolve linters
* fix failing tests and linters
* fix failing test
* fix errors in s3 and sftp
* remove errors from ftp and s3 package
* add tests
* add tests
* update implementation to remove common fields from gcs and other providers to shift to common file system implementation
* restore New method to prevent breaking change
* fix linters
* update docs
---------
Co-authored-by: Suryakant <dassuryakantadas@gmail.com>
docker run -d --name oracle-free -p 1521:1521 -e ORACLE_PWD=YourPasswordHere container-registry.oracle.com/database/free:latest
105
+
docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest
105
106
```
106
107
107
108
> [!NOTE]
@@ -144,4 +145,4 @@ docker run -d --name oracle-free -p 1521:1521 -e ORACLE_PWD=YourPasswordHere con
144
145
- Use trailing white space or the <br> HTML tag at the end of the line.
145
146
- Use "`" sign to add single line code and "```" to add multi-line code block.
146
147
- Use relative references to images (in `public` folder as mentioned above.)
147
-
* The [gofr.dev documentation](https://gofr.dev/docs) site is updated upon push to `/docs` path in the repo. Verify your changes are live after next GoFr version.
148
+
* The [gofr.dev documentation](https://gofr.dev/docs) site is updated upon push to `/docs` path in the repo. Verify your changes are live after next GoFr version.
@@ -6,9 +6,15 @@ GoFr simplifies the complexity of working with different file stores by offering
6
6
7
7
By default, local file-store is initialized and user can access it from the context.
8
8
9
-
GoFr also supports FTP/SFTP file-store. Developers can also connect and use their AWS S3 bucket as a file-store. The file-store can be initialized as follows:
9
+
GoFr also supports FTP/SFTP file-store. Developers can also connect and use their cloud storage bucket as a file-store. Following cloud storage options are currently supported:
10
+
11
+
-**AWS S3**
12
+
-**Google Cloud Storage (GCS)**
13
+
14
+
The file-store can be initialized as follows:
10
15
11
16
### FTP file-store
17
+
12
18
```go
13
19
package main
14
20
@@ -34,6 +40,7 @@ func main() {
34
40
```
35
41
36
42
### SFTP file-store
43
+
37
44
```go
38
45
package main
39
46
@@ -60,8 +67,7 @@ func main() {
60
67
### AWS S3 Bucket as File-Store
61
68
62
69
To run S3 File-Store locally we can use localstack,
63
-
``docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack``
64
-
70
+
`docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack`
65
71
66
72
```go
67
73
package main
@@ -90,17 +96,83 @@ func main() {
90
96
app.Run()
91
97
}
92
98
```
93
-
> Note: The current implementation supports handling only one bucket at a time,
94
-
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.
99
+
100
+
> Note: The current implementation supports handling only one bucket at a time,
101
+
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.
102
+
103
+
### Google Cloud Storage (GCS) Bucket as File-Store
104
+
105
+
To run GCS File-Store locally we can use fake-gcs-server:
106
+
`docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest`
107
+
108
+
```go
109
+
package main
110
+
111
+
import (
112
+
"gofr.dev/pkg/gofr"
113
+
114
+
"gofr.dev/pkg/gofr/datasource/file/gcs"
115
+
)
116
+
117
+
funcmain() {
118
+
app:= gofr.New()
119
+
120
+
// Option 1: Using JSON credentials with local emulator
app.Logger().Fatalf("Failed to initialize GCS: %v", err)
130
+
131
+
}
132
+
133
+
app.AddFileStore(fs)
134
+
135
+
// Option 2: Using default credentials (GOOGLE_APPLICATION_CREDENTIALS)
136
+
// fs, err := gcs.New(&gcs.Config{
137
+
// BucketName: "my-bucket",
138
+
// ProjectID: "my-project-id",
139
+
// }))
140
+
141
+
// Option 3: Direct connection to real GCS (no EndPoint)
142
+
// fs, err := gcs.New(&gcs.Config{
143
+
// BucketName: "my-bucket",
144
+
// CredentialsJSON: readFile("prod-creds.json"),
145
+
// ProjectID: "my-project-id",
146
+
// }))
147
+
148
+
app.Run()
149
+
}
150
+
151
+
// Helper function to read credentials file
152
+
funcreadFile(filenamestring) []byte {
153
+
data, err:= os.ReadFile(filename)
154
+
if err != nil {
155
+
log.Fatalf("Failed to read credentials file: %v", err)
156
+
}
157
+
return data
158
+
}
159
+
160
+
```
161
+
162
+
> **Note:** When connecting to the actual GCS service, authentication can be provided via CredentialsJSON or the GOOGLE_APPLICATION_CREDENTIALS environment variable.
163
+
> When using fake-gcs-server, authentication is not required.
164
+
> Currently supports one bucket per file-store instance.
To switch to another directory in same parent directory
195
+
122
196
```go
123
197
currentDir, err:= ctx.File.Chdir("../my_dir2")
124
198
```
125
199
126
200
To switch to a subfolder of the current directory
201
+
127
202
```go
128
203
currentDir, err:= ctx.File.Chdir("sub_dir")
129
204
```
205
+
130
206
> Note: This method attempts to change the directory, but S3's flat structure and fixed bucket
131
-
> make this operation inapplicable.
207
+
> make this operation inapplicable. Similarly, GCS uses a flat structure where directories are simulated through object prefixes.
132
208
133
209
### Read a Directory
134
210
135
-
The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.
211
+
The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.
136
212
137
213
If an error occurs during the read operation, ReadDir returns the successfully read entries up to the point of the error along with the error itself. Passing "." as the directory argument returns the entries for the current directory.
214
+
138
215
```go
139
216
entries, err:= ctx.File.ReadDir("../testdir")
140
217
@@ -143,12 +220,13 @@ for _, entry := range entries {
143
220
144
221
if entry.IsDir() {
145
222
entryType = "Dir"
146
-
}
223
+
}
147
224
148
225
fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
149
226
}
150
227
```
151
-
> Note: In S3, directories are represented as prefixes of file keys. This method retrieves file
228
+
229
+
> Note: In S3 and GCS, directories are represented as prefixes of file keys/object names. This method retrieves file
152
230
> entries only from the immediate level within the specified directory.
GoFr support reading CSV/JSON/TEXT files line by line.
167
246
168
247
```go
169
248
reader, err:= file.ReadAll()
170
249
171
250
for reader.Next() {
172
251
varbstring
173
-
252
+
174
253
// For reading CSV/TEXT files user need to pass pointer to string to SCAN.
175
254
// In case of JSON user should pass structs with JSON tags as defined in encoding/json.
176
255
err = reader.Scan(&b)
@@ -179,10 +258,12 @@ for reader.Next() {
179
258
}
180
259
```
181
260
182
-
183
261
### Opening and Reading Content from a File
262
+
184
263
To open a file with default settings, use the `Open` command, which provides read and seek permissions only. For write permissions, use `OpenFile` with the appropriate file modes.
264
+
185
265
> Note: In FTP, file permissions are not differentiated; both `Open` and `OpenFile` allow all file operations regardless of specified permissions.
266
+
186
267
```go
187
268
csvFile, _:= ctx.File.Open("my_file.csv")
188
269
@@ -205,6 +286,7 @@ if err != nil {
205
286
### Getting Information of the file/directory
206
287
207
288
Stat retrieves details of a file or directory, including its name, size, last modified time, and type (such as whether it is a file or folder)
289
+
208
290
```go
209
291
file, _:= ctx.File.Stat("my_file.text")
210
292
entryType:="File"
@@ -215,10 +297,12 @@ if entry.IsDir() {
215
297
216
298
fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
217
299
```
218
-
>Note: In S3:
300
+
301
+
> Note: In S3 and GCS:
302
+
>
219
303
> - Names without a file extension are treated as directories by default.
220
-
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed.
221
-
>
304
+
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed (S3 specific behavior).
305
+
>
222
306
> For directories, the method calculates the total size of all contained objects and returns the most recent modification time. For files, it directly returns the file's size and last modified time.
> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method.
321
+
322
+
> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method. GCS supports deletion of both files and empty directories.
323
+
238
324
```go
239
325
err:= ctx.File.Remove("my_dir")
240
326
```
241
327
242
328
The `RemoveAll` command deletes all subdirectories as well. If you delete the current working directory, such as "../currentDir", the working directory will be reset to its parent directory.
243
-
> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided.
329
+
330
+
> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided for S3.
331
+
> GCS handles both files and directories.
332
+
244
333
```go
245
334
err:= ctx.File.RemoveAll("my_dir/my_text")
246
335
```
247
336
248
-
> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 uses
249
-
> a flat file structure, all methods require a full path relative to the S3 bucket.
337
+
> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 and GCS use
338
+
> a flat file structure, all methods require a full path relative to the bucket.
250
339
251
340
> Errors have been skipped in the example to focus on the core logic, it is recommended to handle all the errors.
0 commit comments