Unable to view new .parquet files using the new "Preview" function

[royalh13]

Preflight Checklist

• I have installed the latest version of Storage Explorer.
• I have checked existing resources, including the troubleshooting guide and the release notes.
• I have searched for similar issues.

Storage Explorer Version

1.30.0

Regression From

No response

Architecture

x64

Storage Explorer Build Number

20230609.2

Platform

Windows

OS Version

Windows 10 Enterprise - Version 10.0.19045 Build 19045

Bug Description

I tried to view the contents of multiple .parquet files by using the new "Preview" feature, but Storage Explorer errored out on all the files I tested.

Below is the error text from the 2 types of errors I'm seeing:

{
"name": "TypeError",
"message": "o.buffer.readUInt32LE is not a function",
"stack": "TypeError: o.buffer.readUInt32LE is not a function\n at ZR (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:736826)\n at Object.QR (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:736928)\n at Tn (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:764741)\n at rD (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:766114)\n at async sb (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:765449)\n at async fb (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:767625)\n at async Mu.readRowGroup (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:763489)\n at async Object.next (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:756357)\n at async ParquetParser.getRecords (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:5046)\n at async getParquetData (file:///C:/Users/My.Name/AppData/Local/Programs/Microsoft%20Azure%20Storage%20Explorer/resources/app/node_modules/@storage-explorer/file-preview/dist/src/Panels/ParquetPreviewPanel/index.js:2:6298)"
}

{
"name": null,
"message": ""Invalid parquet type: DECIMAL, for Column: AjeraBilledConsultant\nInvalid parquet type: DECIMAL, for Column: AjeraBilledLabor\nInvalid parquet type: DECIMAL, for Column: AjeraBilledReimbursable\nInvalid parquet type: DECIMAL, for Column: AjeraCostConsultant\nInvalid parquet type: DECIMAL, for Column: AjeraCostLabor\nInvalid parquet type: DECIMAL, for Column: AjeraCostReimbursable\nInvalid parquet type: DECIMAL, for Column: AjeraReceivedConsultant\nInvalid parquet type: DECIMAL, for Column: AjeraReceivedLabor\nInvalid parquet type: DECIMAL, for Column: AjeraReceivedReimbursable\nInvalid parquet type: DECIMAL, for Column: AjeraSpentConsultant\nInvalid parquet type: DECIMAL, for Column: AjeraSpentLabor\nInvalid parquet type: DECIMAL, for Column: AjeraSpentReimbursable\nInvalid parquet type: DECIMAL, for Column: AjeraWIPConsultant\nInvalid parquet type: DECIMAL, for Column: AjeraWIPLabor\nInvalid parquet type: DECIMAL, for Column: AjeraWIPReimbursable\nInvalid parquet type: DECIMAL, for Column: BillingExchangeRate\nInvalid parquet type: DECIMAL, for Column: BudOHRate\nInvalid parquet type: DECIMAL, for Column: ConsultFee\nInvalid parquet type: DECIMAL, for Column: ConsultFeeBillingCurrency\nInvalid parquet type: DECIMAL, for Column: ConsultFeeFunctionalCurrency\nInvalid parquet type: DECIMAL, for Column: ExpPctComp\nInvalid parquet type: DECIMAL, for Column: FEAddlExpenses\nInvalid parquet type: DECIMAL, for Column: FEAddlExpensesPct\nInvalid parquet type: DECIMAL, for Column: Fee\nInvalid parquet type: DECIMAL, for Column: FeeBillingCurrency\nInvalid parquet type: DECIMAL, for Column: FeeDirExp\nInvalid parquet type: DECIMAL, for Column: FeeDirExpBillingCurrency\nInvalid parquet type: DECIMAL, for Column: FeeDirExpFunctionalCurrency\nInvalid parquet type: DECIMAL, for Column: FeeDirLab\nInvalid parquet type: DECIMAL, for Column: FeeDirLabBillingCurrency\nInvalid parquet type: DECIMAL, for Column: FeeDirLabFunctionalCurrency\nInvalid parquet type: DECIMAL, for Column: FeeFunctionalCurrency\nInvalid parquet type: DECIMAL, for Column: FEOther\nInvalid parquet type: DECIMAL, for Column: FEOtherPct\nInvalid parquet type: DECIMAL, for Column: FESurcharge\nInvalid parquet type: DECIMAL, for Column: FESurchargePct\nInvalid parquet type: DECIMAL, for Column: FirmCost\nInvalid parquet type: DECIMAL, for Column: ICBillingExpMult\nInvalid parquet type: DECIMAL, for Column: ICBillingLabMult\nInvalid parquet type: DECIMAL, for Column: LabPctComp\nInvalid parquet type: DECIMAL, for Column: MultAmt\nInvalid parquet type: DECIMAL, for Column: PctComp\nInvalid parquet type: DECIMAL, for Column: POCNSRate\nInvalid parquet type: DECIMAL, for Column: PORMBRate\nInvalid parquet type: DECIMAL, for Column: ProjectExchangeRate\nInvalid parquet type: DECIMAL, for Column: ReimbAllow\nInvalid parquet type: DECIMAL, for Column: ReimbAllowBillingCurrency\nInvalid parquet type: DECIMAL, for Column: ReimbAllowCons\nInvalid parquet type: DECIMAL, for Column: ReimbAllowConsBillingCurrency\nInvalid parquet type: DECIMAL, for Column: ReimbAllowConsFunctionalCurrency\nInvalid parquet type: DECIMAL, for Column: ReimbAllowExp\nInvalid parquet type: DECIMAL, for Column: ReimbAllowExpBillingCurrency\nInvalid parquet type: DECIMAL, for Column: ReimbAllowExpFunctionalCurrency\nInvalid parquet type: DECIMAL, for Column: ReimbAllowFunctionalCurrency\nInvalid parquet type: DECIMAL, for Column: TotalProjectCost\nInvalid parquet type: DECIMAL, for Column: XChargeMult""
}

Steps to Reproduce

Launch Storage Explorer
Drill down to the appropriate folder containing the .parquet files
Select the .parquet file
Push the "Preview" button

Actual Experience

I Launched Storage Explorer.
Then I drilled down to the appropriate folder containing the .parquet file I wanted to preview
Then I selected the .parquet file
Then I pushed the "Preview" button

Expected Experience

I expected a preview of the data within the .parquet file.

Additional Context

No response

[JasonYeMSFT]

The library we use may have limitations and may not support all data types. Let me see if your file runs into a limitation or we have a bug in our code.

Edit: I just checked. We are using a version of the library that doesn't support the DECIMAL data type. A new version of the library added support to it. I will update the library and test it in 1.31.0.
https://github.com/LibertyDSNP/parquetjs/releases/tag/v1.2.3

Also, could you share your data's schema that sees the buffer.readUInt32LE error with us? We would like to know the structure of the data (e.g. field names, nesting structure and data types). You can redact all the actual field names to remove your personal information.

[royalh13]

Table info attached...
Table Info.txt

[mithom]

having a similar issue, here is the error info, if requested i can also share a file
perror.txt

[indexample]

what about other data types? i get a similar issue, but for Double

{
"name": "TypeError",
"message": "o.buffer.readDoubleLE is not a function",
"stack": "TypeError: o.buffer.readDoubleLE is not a function\n at GR

[JasonYeMSFT]

For the readX is not a function type of errors, they are likely due to a bug in the library not converting the uncompressed data to buffer correctly. I'll try to get someone fix it in the library.

[LoHertel]

I got the o.buffer.readUInt32LE is not a function error as well.

The parquet was created from Azure Data Factory (ADF).
The only numeric column is ExtractDate I believe.
According to DuckDB it's of the INT96 type, according to Pyarrow, it's of type timestamp[ns].
The column was exported from the ADF as type DateTime.

DuckDB Schema Information

     file_name      name                        type          type_length  repetition_type    num_children     converted_type    scale    precision    field_id    logical_type
0    abc.parquet    adms_schema                 BOOLEAN       0            REQUIRED           33               UTF8              0        0            0            None
1    abc.parquet    user_id                     BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
2    abc.parquet    displayName                 BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
3    abc.parquet    surname                     BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
4    abc.parquet    givenName                   BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
5    abc.parquet    employeeId                  BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
6    abc.parquet    mail                        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
7    abc.parquet    jobTitle                    BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
8    abc.parquet    department                  BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
9    abc.parquet    companyName                 BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
10   abc.parquet    onPremisesDomainName        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
11   abc.parquet    onPremisesSamAccountName    BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
12   abc.parquet    accountEnabled              BOOLEAN       0            OPTIONAL           0                UTF8              0        0            0            None
13   abc.parquet    userType                    BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
14   abc.parquet    extensionAttribute1         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
15   abc.parquet    extensionAttribute2         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
16   abc.parquet    extensionAttribute3         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
17   abc.parquet    extensionAttribute4         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
18   abc.parquet    extensionAttribute5         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
19   abc.parquet    extensionAttribute6         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
20   abc.parquet    extensionAttribute7         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
21   abc.parquet    extensionAttribute8         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
22   abc.parquet    extensionAttribute9         BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
23   abc.parquet    extensionAttribute10        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
24   abc.parquet    extensionAttribute11        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
25   abc.parquet    extensionAttribute12        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
26   abc.parquet    extensionAttribute13        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
27   abc.parquet    extensionAttribute14        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
28   abc.parquet    extensionAttribute15        BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
29   abc.parquet    costCenter                  BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
30   abc.parquet    division                    BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
31   abc.parquet    manager_id                  BYTE_ARRAY    0            OPTIONAL           0                UTF8              0        0            0            None
32   abc.parquet    ExtractDate                 INT96         0            OPTIONAL           0                UTF8              0        0            0            None

Pyarrow Schema Information

user_id: string
displayName: string
surname: string
givenName: string
employeeId: string
mail: string
jobTitle: string
department: string
companyName: string
onPremisesDomainName: string
onPremisesSamAccountName: string
accountEnabled: bool
userType: string
extensionAttribute1: string
extensionAttribute2: string
extensionAttribute3: string
extensionAttribute4: string
extensionAttribute5: string
extensionAttribute6: string
extensionAttribute7: string
extensionAttribute8: string
extensionAttribute9: string
extensionAttribute10: string
extensionAttribute11: string
extensionAttribute12: string
extensionAttribute13: string
extensionAttribute14: string
extensionAttribute15: string
costCenter: string
division: string
manager_id: string
ExtractDate: timestamp[ns]
-- schema metadata --
writer.model.name: 'example'

ADF Schema of Copy Task

        "translator": {
            "type": "TabularTranslator",
            "mappings": [
                {
                    "source": {
                        "path": "['id']"
                    },
                    "sink": {
                        "name": "user_id",
                        "type": "Guid"
                    }
                },
                {
                    "source": {
                        "path": "['displayName']"
                    },
                    "sink": {
                        "name": "displayName",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['surname']"
                    },
                    "sink": {
                        "name": "surname",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['givenName']"
                    },
                    "sink": {
                        "name": "givenName",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['employeeId']"
                    },
                    "sink": {
                        "name": "employeeId",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['mail']"
                    },
                    "sink": {
                        "name": "mail",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['jobTitle']"
                    },
                    "sink": {
                        "name": "jobTitle",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['department']"
                    },
                    "sink": {
                        "name": "department",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['companyName']"
                    },
                    "sink": {
                        "name": "companyName",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesDomainName']"
                    },
                    "sink": {
                        "name": "onPremisesDomainName",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesSamAccountName']"
                    },
                    "sink": {
                        "name": "onPremisesSamAccountName",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['accountEnabled']"
                    },
                    "sink": {
                        "name": "accountEnabled",
                        "type": "Boolean"
                    }
                },
                {
                    "source": {
                        "path": "['userType']"
                    },
                    "sink": {
                        "name": "userType",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute1']"
                    },
                    "sink": {
                        "name": "extensionAttribute1",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute2']"
                    },
                    "sink": {
                        "name": "extensionAttribute2",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute3']"
                    },
                    "sink": {
                        "name": "extensionAttribute3",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute4']"
                    },
                    "sink": {
                        "name": "extensionAttribute4",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute5']"
                    },
                    "sink": {
                        "name": "extensionAttribute5",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute6']"
                    },
                    "sink": {
                        "name": "extensionAttribute6",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute7']"
                    },
                    "sink": {
                        "name": "extensionAttribute7",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute8']"
                    },
                    "sink": {
                        "name": "extensionAttribute8",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute9']"
                    },
                    "sink": {
                        "name": "extensionAttribute9",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute10']"
                    },
                    "sink": {
                        "name": "extensionAttribute10",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute11']"
                    },
                    "sink": {
                        "name": "extensionAttribute11",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute12']"
                    },
                    "sink": {
                        "name": "extensionAttribute12",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute13']"
                    },
                    "sink": {
                        "name": "extensionAttribute13",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute14']"
                    },
                    "sink": {
                        "name": "extensionAttribute14",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['onPremisesExtensionAttributes']['extensionAttribute15']"
                    },
                    "sink": {
                        "name": "extensionAttribute15",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['employeeOrgData']['costCenter']"
                    },
                    "sink": {
                        "name": "costCenter",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['employeeOrgData']['division']"
                    },
                    "sink": {
                        "name": "division",
                        "type": "String"
                    }
                },
                {
                    "source": {
                        "path": "['manager']['id']"
                    },
                    "sink": {
                        "name": "manager_id",
                        "type": "Guid"
                    }
                },
                {
                    "source": {
                        "path": "$['ExtractDate']"
                    },
                    "sink": {
                        "name": "ExtractDate",
                        "type": "DateTime"
                    }
                }
            ],
        }

[JasonYeMSFT]

@mithom Would you like to share a file with us? Please make sure you don't accidentally share any sensitive information with us. If you don't want to post it here, you can email it to sehelp at microsoft dot com. Thanks.

[mithom]

i will ask my manager, but it shouldn't be hard to generate a file with some noise data in it. We don't handle user data so gdpr wise everything is safe anyways

[tobimax]

Same issue, part of a delta lake created by Databricks using their default writer settings in Unity Catalog Managed Tables.

{
  "name": null,
  "message": "\"Invalid parquet type: DECIMAL, for Column: volume_sold_in_millilitres\\nInvalid parquet type: DECIMAL, for Column: weight_sold_in_grams\\nInvalid parquet type: DECIMAL, for Column: units_sold_by_volume\\nInvalid parquet type: DECIMAL, for Column: units_sold_by_weight\""
}

Attached the data file. there nothing sensitive in it.

part-00004-7690cf81-ffe2-49dc-aaa2-9e622a3b3f5a.c000.snappy.parquet.zip