Add partitions in Hive format on GCP/AWS bucket integration
P
Persimmon Shrimp
current export flow supports
{bucket}/{date}/{table}/{file_name}.parquet
in Redshift or BigQuery it's impossible to use external partitioned tables in current format, there needs to rename/move files to a proper folder structure.
if it would be
{bucket}/{table}/dt={date}/{file_name}.parquet
...
gs://.../person/dt=2025-02-05/part-00...
gs://.../person/dt=2025-03-05/part-00...
gs://.../education/dt=2025-02-05/part-00...
gs://.../education/dt=2025-03-05/part-00...
....
there will be possibility to use a direct files as native Hive partitions.