Idea: performance improvement: Use DuckDB as a intermediate layer.

[matthiasfeist]

DuckDB is exceptionally fast in reading CSV files and it can be used to write data directly into Postgres as well.

For the Node-GTFS project, I ran a quick benchmark here: BlinkTagInc/node-gtfs#148 (comment)
Then using DuckDB, the imported files could be modified and cleaned up directly in DuckDB, and then subsequently exported to Postgres: https://duckdb.org/docs/extensions/postgres.html#writing-data-to-postgresql

[derhuerst]

I'm not sure if the "CSV -> Node -> DuckDB -> PostgreSQL" route is (significantly) faster than the current "CSV -> Node -> PostgreSQL" route, because both the DuckDB CSV import and the PostgreSQL insertion together would have to be (significantly) faster than PostgreSQL's COPY … FROM STDIN csv (e.g. here). The performance gains would have to justify quite an increase in complexity. (Please comment and re-open if you think so!)

Instead, I'd rather investigate if switching to DuckDB altogether is possible by now (help is very welcome!): #39.