9.29.2021 Release Notes
about 3 years ago
⚠️NOTICE⚠️
Ascend will be moving the default Spark runtime to 3.2 now that Spark is getting close to release this version. We will provide exact timing as that week gets closer (in the next few weeks). Given this is a minor version, in our testing we are not seeing any issues / breakages from previous versions of Spark. If for some reason you would like to stay on the current version of Spark, please ensure that your Data Service default is set appropriately to avoid running on the new version of Spark
🚀 FEATURES🚀
- New Apache Druid Write Connector.
- Allow Read Connectors to specify the refresh schedule with Unix Cron Expressions. Allows combining multiple Cron Expressions to create flexible refresh schedules.
- Dynamic Listing and Small File Aggregation for Read Connectors
✨ ENHANCEMENTS✨
Set TCP Keepalive for Legacy Redshift Write Connectors to ensure connections stay open when the Redshift cluster is experiencing high load.
- Add support for a concurrent parameter for the new JDBC write connectors (Postgres, MS SQL, etc). This parameter allows separate Ascend partitions to write concurrently to the target. Note, each individual partition still writes concurrently according the Spark partitions and for now, the new Write Connectors still display as 1 partition.
- Show the full stacktrace when encountering a job terminated error in Spark -- allowing for more quickly finding the root cause.
- Allow nested upstream column types to be written to a MySQL WC with cast to json encoded string.
- When experiencing a Record Loss event in Mongo CDC, update the fields in the fullDocument to more easily be able to view the time range of lost oplog history.
🔧 BUGFIXES🔧
- Handle the case of the Delta Lake Write Connector choosing to register to the AWS Glue Catalog at the connector level without setting up the connection level configuration
- Fix a case in the UI around Legacy S3 Write Connectors giving an error when changing the upstream.
- Fix a case for Ascend GCP deployments and the new BigQuery Write Connector conflicting on credentials.