this post was submitted on 14 Jun 2023
0 points (NaN% liked)

Programming

13371 readers
1 users here now

All things programming and coding related. Subcommunity of Technology.

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 1 year ago
MODERATORS
alyaza@beehaw.org
Hexorg@beehaw.org
remington@beehaw.org
0
Is SQL a good language for data transformations? (discuss.tchncs.de)
submitted 1 year ago by Mattol@discuss.tchncs.de to c/programming@beehaw.org
1 comments fedilink hide all child comments
 

I always wish I wouldn't have to pull in python/pandas to do simple data transformations on smaller local datasets/csv files and stick with sqlite. But then I tend to quickly hit a problem that seems to require dynamic SQL. Anything in between SQL and a general programming language that is made for transforming tabular data?

top 1 comments
sorted by: hot top controversial new old
[–] selawdivad@lemm.ee 1 points 1 year ago

Can I suggest duckdb? You can start out writing SQL directly on top of CSV and Parquet files. But then if you want/need to do something more complicated, you can import duckdb into Python, keep running the SQL you already wrote, convert it to a Pandas or Polars dataframe, transform it, then query the result in SQL.