Custom Partitioning for Parquet Deliveries
W
Wonderful Dolphin
I'm the original requester here. The logic for the request is that PDL "time series" and "headcount summary" tables are the perfect scenario for parquet partitioning, and would significantly reduce the compute needed to analyze subsets of the data.
Notably, my firm's use case is more data science and not simply uploading everything to replica SQL tables. I am our sole data engineer and am aiming to keep processing on a single node.