08.24.2023 Release Notes

These are the release notes for August 24, 2023.

📰 NEWS 📰

• 📧 Coming soon - Email Notifications for dataflow events!
  • You asked for it, and we're currently building and testing email notifications.
  • This will be added as a new option in addition to our existing Webhook notifications feature.
  • Stay tuned to our release notes for the release date!
• A reminder regarding Gen1 to Gen2 migrations:
  • We highly recommend that customers contact Ascend to discuss and schedule migration from Gen1 to Gen2 architecture. Thank you to those customers who have already done so!

✨ FEATURES ✨

• All environments (Gen1/Gen2)
  • Add two additional Data Replication strategies for Blob Read Connectors.
    • When no Data Replication strategy is selected (default):
      • Ascend uses the long-standing and standard replication strategy which ensures the data in Ascend is an exact replica of the data in the Blob storage source (which is maintained on component refresh). If a file is deleted in the source, on the next refresh, it's deleted from Ascend.
    • The two new strategies are:
      • Latest File Versions: Re-ingest every new/modified file in the source. For modified files, the new source file overwrites the previous version in Ascend. If a file is deleted in the source, Ascend keeps the last version of the deleted file.
      • All File Versions: Preserve every version of a file in the source. The partition name includes the last modified timestamp to distinguish between versions, for example: my_prefix/a/b/file1.parquet@2023-01-01T12:00:00. Deleted files and their various versions are also preserved.
      • We anticipate users applying these new strategies when, for example, they want to purge or delete files in the Blob storage source, but keep the ingested data in Ascend. Another use might be to keep all version/change history for files.
      • We will continue to refine these new strategies based on user feedback!
• Gen2 environments
  • All Data Planes
    • Introduce the ability for users to manage container registry credentials required to pull custom Docker images for Ascend Clusters (in Site Admin > Cluster Pools).
      • This option appears in Cluster Pool Management under Runtime Settings - select "CONTAINER IMAGE" = "Container image URL" and then create "REGISTRY CREDENTIALS".

✨ ENHANCEMENTS ✨

• All environments (Gen1/Gen2)
  • Update the Snowflake JDBC driver library to include error messages for failed queries when running on the Ascend Cluster.
    • Ascend worked with the vendor (Snowflake), who has added more information to their JDBC driver library for cases when FAILED_WITH_ERROR messages would occur. This should make it easier to understand what error occurred within the Snowflake platform itself.
    • See related Ascend Support article for context on the previous solution that this change should help address.
• Gen2 environments
  • All Data Planes
    • Implement retry behavior for certain Data Plane commit errors:
      • When Ascend commits/writes data to the Data Plane (e.g. Snowflake, BigQuery, Databricks, Spark w/ Iceberg), there are some intermittent or transient errors that can be retried automatically, without the component having to enter permanent failure state.
      • We've automated retries for these errors, eliminating the need for a user to manually reset errors.

🔧 BUGFIXES 🔧

• All environments (Gen1/Gen2)
  • We now automatically reset commit errors when a component configuration is updated.
• Gen1 environments
  • Address an issue where the Query Service cannot process queries with a large number of input partitions.
• Gen2 environments
  • All Data Planes
    • Fix a bug where task cancellation occasionally causes orphaned tasks for tasks on Ascend Clusters.
    • Resolve a bug causing certain Data Service Data Plane configuration fields to be overridden by default values when the Dataflow Data Plane configuration is set.