Blob Storage Read Connector Parsers

Of the available parsers for some Read Connectors, the following have additional required or optional properties.

CSV Parser

The CSV parser provides a range of customizable parameters. Each parser for a Read Connector has specific parameters.

CSV Parser Fields

All fields available for the CSV parser are optional. We'll use the following text example to discuss each field:

@Album released in 1967 
artist,album,track,verse
The Beatles, Magical Mystery Tour, "Hello, Goodbye", "You say, /"Yes/", I say, /"No/" !/I say, /"Yes/", 
but I may mean, /"No/"!/"

Field	Description
FIELD DELIMITER	Field delimiter to be used. Ex. Delimiter = `,`
FILES HAVE A HEADER ROW	Tells the parser that the first line should be used as column names in the schema. Leave this unchecked if the first row does not contain column names. Ex. Header Row = `artist,album,song,verse`
COMMENT PREFIX	A single character/symbol used for skipping lines beginning with this character. By default, this is disabled. Ex. @Album release in 1967 ( in this example, the Comment prefix is `@`.)
QUOTE CHARACTER	A single character used for escaping quoted values where the separator can be part of the value. If you would like to turn off quotations, you need to set the value to an empty string. Ex. track = "Hello, Goodbye" (In this example, the quote character is `"`.)
ESCAPE CHARACTER	A single character used for escaping quotes inside an already quoted value. Ex. verse1= "You say, /"Yes/", I say, /"No/" !/I say, /"Yes/", but I may mean, /"No/"!/" (In this example, the escape character is `/`.)
ENABLE MULTILINE VARIABLES	This checkbox informs the parser that there can be multi-line fields in the ingested data. In most CSV files, each record stands on its own line, with each field separated by a comma. But sometimes we have a multi-line field. Ex. verse1= "You say, /"Yes/", I say, /"No/" !/I say, /"Yes/", but I may mean, /"No/"!/" (In this example, the verse is continued on a new line.)
DATE FORMAT	Format of the date type field in the CSV file. See Datetime Patterns for Formatting and Parsing.
TIMESTAMP FORMAT	Format of the timestamp type field in the CSV file. Specify the timestamp format if known. See Datetime Patterns for Formatting and Parsing.
ENCODING	Decode CSV files by the given encoding type. Defaults to UTF-8 (this is the most common encoding type). One known exception is files created on Windows, which requires `UTF-16`, instead.
LINE SEPARATOR	Defines the line separator that should be used for parsing. Defaults to `\n` (the newline character).
CHAR TO ESCAPE QUOTE ESCAPING	Sets a single character used for escaping the escape for the quote character. Ex. verse1= "You say, /"Yes/", I say, /"No/" !/I say, /"Yes/", but I may mean, /"No/"!/"* (In this example, the lyrics utilize a forward slash to denote the addition of background lyrics. Adding an exclamation mark (!) before the forward slash will preserve the forward slash in the string. The character to escape quote escaping is `!`.)
MODE	Allows a mode for dealing with corrupt records during parsing. Mode options include PERMISSIVE, DROPMALFORMED, and FAILFAST. PERMISSIVE: when meeting a corrupted record, puts the malformed string into a field configured by _corrupt_record, and sets malformed fields to null. To keep corrupt records, a user can set a string type field named _corrupt_record in a user-defined schema. If a schema does not have the field, it drops corrupt records during parsing. DROPMALFORMED: ignores the whole corrupted records. This mode is unsupported in the CSV built-in functions. FAILFAST: throws an exception when meeting corrupted records.

Excel Parser

All fields for the Excel parser are optional.

Excel Parser Fields

Field	Description
NUMBER OF HEADER ROWS	Indicates the number of header rows used in the sheet used for parsing. If no number of rows is indicated, Ascend assumes the first row is the header row.
SHEET NAME	Indicates the name of the sheet to use. If no sheet is selected, the first sheet of the Excel file will be used.

JSON Parser

All fields for the JSON parser are optional.

JSON Parser Fields

Field	Required
DATA FORMAT	Format of the date type field in the CSV file. See Datetime Patterns for Formatting and Parsing.
TIMESTAMP FORMAT	Format of the timestamp type field in the CSV file. Specify the timestamp format if known. See Datetime Patterns for Formatting and Parsing.

Python Parser

Python Parser Fields

Field	Required	Description
PARSER CODE INTERFACE	Required	Current availability is Byte Stream with Callback Function
CODE FINGERPRINT STRATEGY	Required	Current availability is Automatic content-based fingerprint
CODE	Required	See below for a code sample highlighting the required parameters.
ADDITIONAL PIP PACKAGES TO INSTALL	Optional	Indicate if there are any additional PIP packages.

Below is a code sample included within Ascend's Python Parser Function.

# required interface parser_function
def parser_function(reader, on_next):
  byte = reader.read(1024)
  while byte != b"":
    # callback with parsed record
    on_next({'chunk': byte.hex()})
    # handle byte, and read next chunk:
    byte = reader.read(1024)