Release Notes
Developer HubDocsSDK
Back to All

06.21.2023 Release Notes

over 1 year ago

These are the release notes for June 21, 2023.

✨ ENHANCEMENTS ✨

  • All environments (Gen1/Gen2)
    • Added support for an optional infer_schema method in Custom Python Read Connector. This allows for overriding the default schema inference process
    • Relocated the Intercom portal icon to the right side of the screen so it no longer covers the buttons.
  • Gen2 Environments:
    • Ascend Clusters based on Spark v3.4.0 are now available in Gen2 environments. This requires an environment configuration change by the Ascend team (submit a request to the support team), which will be necessary until the custom docker images feature is released.
    • Enhanced overall execution performance when running numerous concurrent tasks by disabling Task Speculation on the Ascend Cluster.
    • Snowflake Data Plane
      • Labels have been added to Snowflake Queries along with Metadata.
    • Iceberg Data Plane:
      • Improved performance of Iceberg Write Connectors and the Iceberg Data Plane by removing previous table states.
      • Optimized the commit step of Iceberg write operations for better performance.

🔧 BUGFIXES 🔧

  • All environments (Gen1/Gen2)
    • Removed "Refresh Now" option for components which cannot be refreshed.
    • Fixed a race condition causing intermittent ModuleNotFound errors for custom Python UDF files.
  • Gen2 environments
    • Improved the stability of the Data Plane Manager by effectively controlling Ascend Clusters in unexpected scenarios, which could have led to tasks remaining in a running state indefinitely.
    • Fixed a case where Spark tasks would remain in a running state due to improper handling of 404 errors during cluster deletion.
    • "Generate Schema", "Test Connection", and "List Asset" requests are now correctly sent to the interactive cluster.
    • Snowflake Data Plane
      • Resolved a "View too large" error on full reductions that occurred when the input component had over approximately 2800 partitions.
    • BigQuery Data Plane
      • Fixed a bug in BQ My Tables where __NULL__ partitions were not correctly handled for range partitioned tables.
    • Iceberg Data Plane
      • Fix a bug in the sweep stage of Iceberg Data Plane that prevented old partitions from being deleted.