You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/docs/learn/parser_in_rust/ast.md
+49-49Lines changed: 49 additions & 49 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -150,6 +150,55 @@ Take a look at the [real world implementation from swc](https://github.com/swc-p
150
150
we can see that an AST can have lots of `Box`s and `Vec`s, and also note that the `Statement` and `Expression` enums contain
151
151
a dozen of enum variants.
152
152
153
+
### Memory Arena
154
+
155
+
Using the global memory allocator for the AST is actually not really efficient.
156
+
Every `Box` and `Vec` are allocated on demand and then dropped separately.
157
+
What we would like to do is pre-allocate memory and drop it in wholesale.
158
+
159
+
:::info
160
+
See also [Arenas in Rust](https://manishearth.github.io/blog/2021/03/15/arenas-in-rust) and [Flattening ASTs](https://www.cs.cornell.edu/~asampson/blog/flattening.html) for more background on storing ASTs in memory arenas.
161
+
:::
162
+
163
+
[`bumpalo`](https://docs.rs/bumpalo/latest/bumpalo/) is a very good candidate for our use case, according to its documentation:
164
+
165
+
> Bump allocation is a fast, but limited approach to allocation.
166
+
> We have a chunk of memory, and we maintain a pointer within that memory. Whenever we allocate an object,
167
+
> we do a quick check that we have enough capacity left in our chunk to allocate the object and then update the pointer by the object’s size. That’s it!
168
+
>
169
+
> The disadvantage of bump allocation is that there is no general way to deallocate individual objects or reclaim the memory region for a no-longer-in-use object.
170
+
>
171
+
> These trade offs make bump allocation well-suited for phase-oriented allocations. That is, a group of objects that will all be allocated during the same program phase, used, and then can all be deallocated together as a group.
172
+
173
+
By using `bumpalo::collections::Vec` and `bumpalo::boxed::Box`, our AST will have lifetimes added to it:
174
+
175
+
```rust
176
+
usebumpalo::collections::Vec;
177
+
usebumpalo::boxed::Box;
178
+
179
+
pubenumExpression<'a> {
180
+
AwaitExpression(Box<'a, AwaitExpression>),
181
+
YieldExpression(Box<'a, YieldExpression>),
182
+
}
183
+
184
+
pubstructAwaitExpression<'a> {
185
+
pubnode:Node,
186
+
pubexpression:Expression<'a>,
187
+
}
188
+
189
+
pubstructYieldExpression<'a> {
190
+
pubnode:Node,
191
+
pubexpression:Expression<'a>,
192
+
}
193
+
```
194
+
195
+
:::info
196
+
Please be cautious if we are not comfortable dealing with lifetimes at this stage.
197
+
Our program will work fine without a memory arena.
198
+
199
+
Code in the following chapters does not demonstrate the use of a memory arena for simplicity.
200
+
:::
201
+
153
202
### Enum Size
154
203
155
204
The first optimization we are going to make is to reduce the size of the enums.
Using the global memory allocator for the AST is actually not really efficient.
248
-
Every `Box` and `Vec` are allocated on demand and then dropped separately.
249
-
What we would like to do is pre-allocate memory and drop it in wholesale.
250
-
251
-
:::info
252
-
See also [Arenas in Rust](https://manishearth.github.io/blog/2021/03/15/arenas-in-rust) and [Flattening ASTs](https://www.cs.cornell.edu/~asampson/blog/flattening.html) for more background on storing ASTs in memory arenas.
253
-
:::
254
-
255
-
[`bumpalo`](https://docs.rs/bumpalo/latest/bumpalo/) is a very good candidate for our use case, according to its documentation:
256
-
257
-
> Bump allocation is a fast, but limited approach to allocation.
258
-
> We have a chunk of memory, and we maintain a pointer within that memory. Whenever we allocate an object,
259
-
> we do a quick check that we have enough capacity left in our chunk to allocate the object and then update the pointer by the object’s size. That’s it!
260
-
>
261
-
> The disadvantage of bump allocation is that there is no general way to deallocate individual objects or reclaim the memory region for a no-longer-in-use object.
262
-
>
263
-
> These trade offs make bump allocation well-suited for phase-oriented allocations. That is, a group of objects that will all be allocated during the same program phase, used, and then can all be deallocated together as a group.
264
-
265
-
By using `bumpalo::collections::Vec` and `bumpalo::boxed::Box`, our AST will have lifetimes added to it:
266
-
267
-
```rust
268
-
usebumpalo::collections::Vec;
269
-
usebumpalo::boxed::Box;
270
-
271
-
pubenumExpression<'a> {
272
-
AwaitExpression(Box<'a, AwaitExpression>),
273
-
YieldExpression(Box<'a, YieldExpression>),
274
-
}
275
-
276
-
pubstructAwaitExpression<'a> {
277
-
pubnode:Node,
278
-
pubexpression:Expression<'a>,
279
-
}
280
-
281
-
pubstructYieldExpression<'a> {
282
-
pubnode:Node,
283
-
pubexpression:Expression<'a>,
284
-
}
285
-
```
286
-
287
-
:::info
288
-
Please be cautious if we are not comfortable dealing with lifetimes at this stage.
289
-
Our program will work fine without a memory arena.
290
-
291
-
Code in the following chapters does not demonstrate the use of a memory arena for simplicity.
292
-
:::
293
-
294
294
## JSON Serialization
295
295
296
296
[serde](https://serde.rs/) can be used serialize the AST to JSON. Some techniques are needed to make it `estree` compatible.
0 commit comments