Release 428 (4 Oct 2023)#
General#
- Reduce memory usage for queries involving - GROUP BYclauses. (#19187)
- Simplify writer count configuration. Add the new - task.min-writer-countand- task.max-writer-countconfiguration properties along with the- task_min_writer_countand- task_max_writer_countsession properties, which control the number of writers depending on scenario. Deprecate the- task.writer-count,- task.scale-writers.max-writer-count, and- task.partitioned-writer-countconfiguration properties, which will be removed in the future. Remove the- task_writer_count,- task_partitioned_writer_count, and- task_scale_writers_max_writer_countsession properties. (#19135)
- Remove support for the - parse-decimal-literals-as-doublelegacy configuration property. (#19166)
- Fix out of memory error when running queries with - GROUP BYclauses. (#19119)
Delta Lake connector#
- Reduce the number of read requests for scanning small Parquet files. Add the - parquet.small-file-thresholdconfiguration property and the- parquet_small_file_thresholdsession property to change the default size of- 3MB, below which, files will be read in their entirety. Setting this configuration to- 0Bdisables the feature. (#19127)
- Fix potential data duplication when running - OPTIMIZEcoincides with updates to a table. (#19128)
- Fix error when deleting rows in tables that have partitions with certain non-alphanumeric characters in their names. (#18922) 
Hive connector#
- Reduce the number of read requests for scanning small Parquet files. Add the - parquet.small-file-thresholdconfiguration property and the- parquet_small_file_thresholdsession property to change the default size of- 3MB, below which, files will be read in their entirety. Setting this configuration to- 0Bdisables the feature. (#19127)
Hudi connector#
- Reduce the number of read requests for scanning small Parquet files. Add the - parquet.small-file-thresholdconfiguration property and the- parquet_small_file_thresholdsession property to change the default size of- 3MB, below which, files will be read in their entirety. Setting this configuration to- 0Bdisables the feature. (#19127)
Iceberg connector#
- Reduce the number of read requests for scanning small Parquet files. Add the - parquet.small-file-thresholdconfiguration property and the- parquet_small_file_thresholdsession property to change the default size of- 3MB, below which, files will be read in their entirety. Setting this configuration to- 0Bdisables the feature. (#19127)
- Fix incorrect column statistics for the Parquet file format in manifest files. (#19052) 
Pinot connector#
- Add support for query options in dynamic tables. (#19078)