06.21.2023 Release Notes

These are the release notes for June 21, 2023.

✨ ENHANCEMENTS ✨

• All environments (Gen1/Gen2)
  • Added support for an optional infer_schema method in Custom Python Read Connector. This allows for overriding the default schema inference process
  • Relocated the Intercom portal icon to the right side of the screen so it no longer covers the buttons.
• Gen2 Environments:
  • Ascend Clusters based on Spark v3.4.0 are now available in Gen2 environments. This requires an environment configuration change by the Ascend team (submit a request to the support team), which will be necessary until the custom docker images feature is released.
  • Enhanced overall execution performance when running numerous concurrent tasks by disabling Task Speculation on the Ascend Cluster.
  • Snowflake Data Plane
    • Labels have been added to Snowflake Queries along with Metadata.
  • Iceberg Data Plane:
    • Improved performance of Iceberg Write Connectors and the Iceberg Data Plane by removing previous table states.
    • Optimized the commit step of Iceberg write operations for better performance.

🔧 BUGFIXES 🔧

• All environments (Gen1/Gen2)
  • Removed "Refresh Now" option for components which cannot be refreshed.
  • Fixed a race condition causing intermittent ModuleNotFound errors for custom Python UDF files.
• Gen2 environments
  • Improved the stability of the Data Plane Manager by effectively controlling Ascend Clusters in unexpected scenarios, which could have led to tasks remaining in a running state indefinitely.
  • Fixed a case where Spark tasks would remain in a running state due to improper handling of 404 errors during cluster deletion.
  • "Generate Schema", "Test Connection", and "List Asset" requests are now correctly sent to the interactive cluster.
  • Snowflake Data Plane
    • Resolved a "View too large" error on full reductions that occurred when the input component had over approximately 2800 partitions.
  • BigQuery Data Plane
    • Fixed a bug in BQ My Tables where __NULL__ partitions were not correctly handled for range partitioned tables.
  • Iceberg Data Plane
    • Fix a bug in the sweep stage of Iceberg Data Plane that prevented old partitions from being deleted.