Skip to content

Commit 2622ff1

Browse files
j4ys0nwilwade
andauthored
allow typeLength to come from opts.column when decoding FIXED_LEN_BYTE_ARRAY (#108)
Problem ======= typeLength is present in column options but decoding is throwing an error. `thrown: "missing option: typeLength (required for FIXED_LEN_BYTE_ARRAY)"` options object for reference: ``` { type: 'FIXED_LEN_BYTE_ARRAY', rLevelMax: 0, dLevelMax: 1, compression: 'SNAPPY', column: { name: 'BLOCK_NUMBER', primitiveType: 'FIXED_LEN_BYTE_ARRAY', originalType: 'DECIMAL', path: [ 'BLOCK_NUMBER' ], repetitionType: 'OPTIONAL', encoding: 'PLAIN', statistics: undefined, compression: 'UNCOMPRESSED', precision: 38, scale: 0, typeLength: 16, rLevelMax: 0, dLevelMax: 1 }, num_values: { buffer: <Buffer 00 00 00 00 00 00 27 10>, offset: 0 } } ``` using `parquet-tools schema` here is the schema for this column: ``` optional fixed_len_byte_array(16) BLOCK_NUMBER (DECIMAL(38,0)) ``` The parquet file is a direct export from snowflake and the data type of the column is `NUMBER(38,0)`. Solution ======== I traced through the code to find where the decode was erroring and added the ability to take the `typeLength` from `column` in the column options when it is not present at the top level. Change summary: --------------- see above Steps to Verify: ---------------- decode a parquet file with this type of field. --------- Co-authored-by: Wil Wade <[email protected]> Co-authored-by: Wil Wade <[email protected]>
1 parent 8d34ac1 commit 2622ff1

File tree

2 files changed

+5
-6
lines changed

2 files changed

+5
-6
lines changed

lib/codec/plain.ts

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -264,16 +264,17 @@ function decodeValues_FIXED_LEN_BYTE_ARRAY(
264264
opts: Options
265265
) {
266266
let values = [];
267-
268-
if (!opts.typeLength) {
267+
const typeLength =
268+
opts.typeLength ?? (opts.column ? opts.column.typeLength : undefined);
269+
if (!typeLength) {
269270
throw "missing option: typeLength (required for FIXED_LEN_BYTE_ARRAY)";
270271
}
271272

272273
for (let i = 0; i < count; ++i) {
273274
values.push(
274-
cursor.buffer.slice(cursor.offset, cursor.offset + opts.typeLength)
275+
cursor.buffer.slice(cursor.offset, cursor.offset + typeLength)
275276
);
276-
cursor.offset += opts.typeLength;
277+
cursor.offset += typeLength;
277278
}
278279

279280
return values;

test/reference-test/read-all.test.ts

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,6 @@ const unsupported = [
2424
'delta_encoding_optional_column.parquet', // DELTA_BINARY_PACKED unsupported
2525
'delta_encoding_required_column.parquet', // DELTA_BINARY_PACKED unsupported
2626
'delta_length_byte_array.parquet', // ZSTD unsupported, DELTA_BINARY_PACKED unsupported
27-
'float16_nonzeros_and_nans.parquet', // missing option: typeLength (required for FIXED_LEN_BYTE_ARRAY)
28-
'float16_zeros_and_nans.parquet', // missing option: typeLength (required for FIXED_LEN_BYTE_ARRAY)
2927
'large_string_map.brotli.parquet', // BUG?
3028
];
3129

0 commit comments

Comments
 (0)