DocParser

DocParser extracts UTF-8 text blocks and basic structural information from various document files in their original order. It supports PDF, DOCX, PPTX, XLSX, and plain text files, and is suitable for scenarios such as document ingestion, full-text search preprocessing, and RAG data preparation. The RAG feature of DolphinX relies on this plugin for file parsing. Load this plugin before using related features. The following document formats are currently supported:

Format Common Extensions Extracted Content Limitations Resource Limits

PDF

.pdf

PDF text layer and page numbers

OCR is not supported; encrypted PDFs are not supported.

  • Single file or BLOB: 100 MiB

  • PDF pages or PPTX slides: 2,000

  • Extracted Unicode characters: 50,000,000

  • Single XML part: 64 MiB

  • XML nesting depth: 256

DOCX

.docx

Headings, paragraphs, lists, tables

Headers and footers are not extracted; nested tables are expanded with a warning.

PPTX

.pptx

Headings, text boxes, tables, slide numbers, speaker notes

Object reading order is inferred from geometric positions; complex layouts may produce warnings.

XLSX

.xlsx

Sheet names, cell stored values row by row

Styles are not processed, formulas are not calculated, and .xls and .xlsm are not supported.

Plain text

.txt, .text, .log, .md, .csv, .tsv

Text paragraphs separated by blank lines

Non-UTF-8 files without BOM must explicitly specify an encoding.

Installation

Version Requirements

Required server version: 3.00.6 and later. Supports Linux ABI.

Installation Steps

  1. In a DolphinDB client, use the listRemotePlugins function to view the plugins available for installation.

    login("admin", "123456")
listRemotePlugins()

  2. Use the installPlugin function to install the plugin.

    installPlugin("DocParser")

  3. Use the loadPlugin function to load the plugin.

    loadPlugin("DocParser")

Method References

parseFile

Syntax

DocParser::parseFile(path, [options])

Details

Parses a specified local file on the DolphinDB server.

Parameters

path A STRING scalar specifying the absolute path to the file on the server where DolphinDB resides. It cannot be empty. The specified file must exist, must be a regular file (not encrypted or corrupted), and the DolphinDB server must have permission to read it.

options (Optional) A dictionary (Dictionary<STRING, ANY>) specifying parsing options. If not specified, default values are used. Key names are case-sensitive; key values are case-insensitive. The following keys are supported:

Key Allowed Values Default Value Description

"format"

  • "auto"

  • "pdf"

  • "docx"

  • "pptx"

  • "xlsx"

  • "text"

"auto"

Specifies the format of the file to parse. If set to "auto", the plugin automatically selects the parser based on the file extension; other values require the extension to map to the same format.

"encoding"

  • "auto"

  • "UTF-8"

  • "UTF-16LE"

  • "UTF-16BE"

  • "UTF-32LE"

  • "UTF-32BE"

  • "GB18030"

"auto"

Specifies the encoding for plain texts. You may omit the hyphen for UTF encoding names.

  • If set to "auto", the plugin detects UTF-8, UTF-16, and UTF-32 based on the UTF BOM.

  • Without a BOM, the input must be valid UTF-8.

  • GB18030 is not auto-detected; it must be explicitly specified.

  • An error is raised if the explicit encoding conflicts with the BOM, or if the byte sequence is invalid.

Returns

Returns a dictionary (Dictionary<STRING, ANY>) containing the following keys:

Key Type Description

"schemaVersion"

STRING

Returns the structure version. Currently "1.0".

"status"

STRING

"OK" or "PARTIAL".

"format"

STRING

The actual parsed format: pdf, docx, pptx, xlsx, or text.

"metadata"

DICTIONARY

Input file and parser metadata. See the metadata dictionary for details.

"warnings"

TABLE

Non-fatal parsing warnings. Returns a zero-row table when there are no warnings. See the warnings table for details.

"blocks"

TABLE

Text blocks in document order. Returns a zero-row table when there is no text. See the blocks table for details.

A "status" of "PARTIAL" indicates that at least part of the document content, pages, character mappings, or optional OOXML parts could not be extracted. In this case, the function still returns normally; applications can use the extracted "blocks" while recording and handling the "warnings".

The presence of "warnings" does not necessarily mean that "status" is "PARTIAL". For example, mismatched extensions or uncertain PPTX reading order may produce "warnings" while "status" remains "OK".

metadata Dictionary

Key Type Description

"fileName"

STRING

The logical file name passed to parseBytes, or the file name portion of the path passed to parseFile.

"fileSize"

LONG

Byte count of the input files.

"parserVersion"

STRING

Parser implementation version.

"sourceEncoding"

STRING

Actual encoding of plain text; NULL for non-text formats.

"pageCount"

INT

PDF page count, PPTX slide count, or XLSX sheet count; null for DOCX and plain text.

warnings Table

Field Type Description

code

SYMBOL

Warning code. Warning codes that may currently be returned:

  • "FORMAT_EXTENSION_MISMATCH" (the file extension does not match the detected content format)

  • "UNSUPPORTED_ELEMENT_SKIPPED" (a page, OOXML part, or structure that could not be fully represented was skipped or expanded)

  • "UNICODE_MAPPING_MISSING" (PDF characters are missing valid Unicode mappings)

  • "POSSIBLE_SCANNED_PDF" (the PDF contains images but little extractable text; OCR may be required)

  • "READING_ORDER_UNCERTAIN" (the PPTX layout is complex, and the reading order of text boxes can only be inferred from their positions)

  • "FORMULA_CACHE_MISSING" (the XLSX formula has no cached result, so the cell is output as null)

message

STRING

Human-readable details.

pageNo

INT

1-based PDF page, PPTX slide, or XLSX sheet number.

sourcePart

STRING

OOXML part name or other source location.

blocks Table

Field Type Description

ordinal

LONG

0-based global text block order.

blockType

SYMBOL

Semantic type of the text block:

  • "pageText" (non-empty extractable text from a PDF page)

  • "heading" (DOCX heading; headingLevel is also provided)

  • "paragraph" (paragraph in DOCX, PPTX, or plain text)

  • "listItem" (bulleted or numbered list item in DOCX)

  • "tableRow" (table row in DOCX, PPTX, or XLSX; cells are separated by tabs)

  • "title" (PPTX title placeholder, or XLSX sheet name)

  • "note" (PPTX single-slide speaker notes)

text

STRING

Normalized UTF-8 text.

pageNo

INT, can be NULL.

1-based PDF page, PPTX slide, or XLSX sheet number.

headingLevel

INT, can be NULL.

1-based DOCX heading level.

groupType

SYMBOL, can be NULL.

Main structure types: "table", "list", "textBox", "note".

groupId

LONG, can be NULL.

Unique structure group identifier within the parse result.

groupOrdinal

INT, can be NULL.

0-based order of text blocks within the group.

groupLevel

INT, can be NULL.

0-based DOCX list nesting level; NULL for other structures.

groupType, groupId, and groupOrdinal are either all present or all NULLs. groupId is only suitable for determining whether two text blocks belong to the same structure within a single parse result; it is not guaranteed to remain stable across calls. To restore document order, use the global ordinal; you must not sort by groupId.

Example

// Simple example
options = dict(["format"], ["docx"])
result = DocParser::parseFile("/data/contracts/contract.docx", options)
blocks = result["blocks"]
// Automatically detect format and text encoding.
autoResult = DocParser::parseFile("/data/readme.txt")
// Explicitly specify GB18030.
gbOptions = dict(["format", "encoding"], ["text", "GB18030"])
gbResult = DocParser::parseFile("/data/legacy.txt", gbOptions)
// Force PDF content validation; raise an error if the file is not a PDF.
pdfOptions = dict(["format"], ["pdf"])
pdfResult = DocParser::parseFile("/data/report.pdf", pdfOptions)

parseBytes

Syntax

DocParser::parseBytes(content, fileName, [options])

Details

Parses a BLOB document in memory. Use this when DolphinDB server has already read the file content, or when the file is not located on the same server as DolphinDB server. When uploading PDF, DOCX, PPTX, or XLSX files, preserve the original binary bytes; do not decode them as strings first.

Parameters

content A BLOB scalar specifying the complete raw document bytes to parse.

fileName A STRING scalar specifying the logical file name, used for returned metadata, extension checking, and diagnostics; it is not opened as a local path.

options (Optional) A dictionary (Dictionary<STRING, ANY>) specifying parsing options. If not specified, default values are used. Key names are case-sensitive; key values are case-insensitive. The following keys are supported:

Key Allowed Values Default Value Description

"format"

  • "auto"

  • "pdf"

  • "docx"

  • "pptx"

  • "xlsx"

  • "text"

"auto"

Specifies the format of the file to parse. If set to "auto", the plugin automatically selects the parser based on the file extension; other values require the extension to map to the same format.

"encoding"

  • "auto"

  • "UTF-8"

  • "UTF-16LE"

  • "UTF-16BE"

  • "UTF-32LE"

  • "UTF-32BE"

  • "GB18030"

"auto"

Specifies the encoding for plain texts. You may omit the hyphen for UTF encoding names.

  • If set to "auto", the plugin detects UTF-8, UTF-16, and UTF-32 based on the UTF BOM.

  • Without a BOM, the input must be valid UTF-8.

  • GB18030 is not auto-detected; it must be explicitly specified.

  • An error is raised if the explicit encoding conflicts with the BOM, or if the byte sequence is invalid.

Returns

Returns a dictionary (Dictionary<STRING, ANY>) containing the following keys:

Key Type Description

"schemaVersion"

STRING

Returns the structure version. Currently "1.0".

"status"

STRING

"OK" or "PARTIAL".

"format"

STRING

The actual parsed format: pdf, docx, pptx, xlsx, or text.

"metadata"

DICTIONARY

Input file and parser metadata. See the metadata dictionary for details.

"warnings"

TABLE

Non-fatal parsing warnings. Returns a zero-row table when there are no warnings. See the warnings table for details.

"blocks"

TABLE

Text blocks in document order. Returns a zero-row table when there is no text. See the blocks table for details.

A "status" of "PARTIAL" indicates that at least part of the document content, pages, character mappings, or optional OOXML parts could not be extracted. In this case, the function still returns normally; applications can use the extracted "blocks" while recording and handling the "warnings".

The presence of "warnings" does not necessarily mean that "status" is "PARTIAL". For example, mismatched extensions or uncertain PPTX reading order may produce "warnings" while "status" remains "OK".

metadata Dictionary

Key Type Description

"fileName"

STRING

The logical file name passed to parseBytes, or the file name portion of the path passed to parseFile.

"fileSize"

LONG

Byte count of the input files.

"parserVersion"

STRING

Parser implementation version.

"sourceEncoding"

STRING

Actual encoding of plain text; NULL for non-text formats.

"pageCount"

INT

PDF page count, PPTX slide count, or XLSX sheet count; null for DOCX and plain text.

warnings Table

Field Type Description

code

SYMBOL

Warning code. Warning codes that may currently be returned:

  • "FORMAT_EXTENSION_MISMATCH" (the file extension does not match the detected content format)

  • "UNSUPPORTED_ELEMENT_SKIPPED" (a page, OOXML part, or structure that could not be fully represented was skipped or expanded)

  • "UNICODE_MAPPING_MISSING" (PDF characters are missing valid Unicode mappings)

  • "POSSIBLE_SCANNED_PDF" (the PDF contains images but little extractable text; OCR may be required)

  • "READING_ORDER_UNCERTAIN" (the PPTX layout is complex, and the reading order of text boxes can only be inferred from their positions)

  • "FORMULA_CACHE_MISSING" (the XLSX formula has no cached result, so the cell is output as null)

message

STRING

Human-readable details.

pageNo

INT

1-based PDF page, PPTX slide, or XLSX sheet number.

sourcePart

STRING

OOXML part name or other source location.

blocks Table

Field Type Description

ordinal

LONG

0-based global text block order.

blockType

SYMBOL

Semantic type of the text block:

  • "pageText" (non-empty extractable text from a PDF page)

  • "heading" (DOCX heading; headingLevel is also provided)

  • "paragraph" (paragraph in DOCX, PPTX, or plain text)

  • "listItem" (bulleted or numbered list item in DOCX)

  • "tableRow" (table row in DOCX, PPTX, or XLSX; cells are separated by tabs)

  • "title" (PPTX title placeholder, or XLSX sheet name)

  • "note" (PPTX single-slide speaker notes)

text

STRING

Normalized UTF-8 text.

pageNo

INT, can be NULL.

1-based PDF page, PPTX slide, or XLSX sheet number.

headingLevel

INT, can be NULL.

1-based DOCX heading level.

groupType

SYMBOL, can be NULL.

Main structure types: "table", "list", "textBox", "note".

groupId

LONG, can be NULL.

Unique structure group identifier within the parse result.

groupOrdinal

INT, can be NULL.

0-based order of text blocks within the group.

groupLevel

INT, can be NULL.

0-based DOCX list nesting level; NULL for other structures.

groupType, groupId, and groupOrdinal are either all present or all NULLs. groupId is only suitable for determining whether two text blocks belong to the same structure within a single parse result; it is not guaranteed to remain stable across calls. To restore document order, use the global ordinal; you must not sort by groupId.

Example

Plain-text BLOB example:

content = blob("first paragraph\n\n second paragraph")
options = dict(["format", "encoding"], ["text", "UTF-8"])
result = DocParser::parseBytes(content, "note.txt", options)

fileNameonly represents a logical name, so the following call does not access /client/path/:

result = DocParser::parseBytes(content, "/client/path/report.pdf")

formats

Syntax

DocParser::formats()

Details

Queries the document formats supported by this plugin.

Parameters

None

Returns

Returns a table with the following fields:

Field Type Description

format

SYMBOL

Document formats supported by this plugin; format names that can be used with options.format.

extensions

STRING

Recognized file extensions; multiple extensions are separated by commas.

mimeTypes

STRING

The corresponding MIME type.

features

STRING

The extraction capabilities supported by the format.

Example

DocParser::formats()

Returns a table:

format extensions mimeTypes features

pdf

.pdf

application/pdf

text,page

docx

.docx

application/vnd.openxmlformats-officedocument.wordprocessingml.document

text,heading,list,table,structureGroup

pptx

.pptx

application/vnd.openxmlformats-officedocument.presentationml.presentation

text,page,title,table,textBox,note,structureGroup

xlsx

.xlsx

application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

text,sheet,title,table,rawValue,structureGroup

text

.txt,.text,.log,.md,.csv,.tsv

text/plain

text,encoding

version

Syntax

DocParser::version()

Details

Returns the plugin version, build environment, and dependency versions.

Parameters

None

Returns

Returns a dictionary (Dictionary<STRING, ANY>) containing the following keys:

Key Description

"pluginVersion"

The DocParser plugin version.

"schemaVersion"

The version of the parsing result structure.

"architecture"

The build target architecture.

"compiler"

The compiler version.

"cppAbi"

The libstdc++ C++ ABI used by the plugin.

"dependencies"

A dictionary containing the versions of PDFium, libzip, pugixml, ICU, and zlib.

If PDFium fails to load, DocParser::version() does not fail entirely; instead, it reports an unavailable status in the PDFium version field.

Examples

// Automatically detect the format and text encoding to parse a pptx file.
result = DocParser::parseFile("/home/test/test.pptx")
blocks = result["blocks"]

// Pass options to specify format as text.
options = dict(["format"], ["text"])
result = DocParser::parseFile("/home/test/test.txt", options)
blocks = result["blocks"]

// Explicitly specify GB18030.
gbOptions = dict(["format", "encoding"], ["text", "GB18030"])
gbResult = DocParser::parseFile("/home/test/test.txt", gbOptions)

// Force validation of the file and raise an error if it is not a PDF.
pdfOptions = dict(["format"], ["pdf"])
pdfResult = DocParser::parseFile("/home/test/test.txt", pdfOptions)
// Error: [DocParser:INVALID_OPTION] options.format 'pdf' does not match file extension '.txt'

Error Code

Throws a DolphinDB runtime exception and does not return a partial dictionary on argument errors or fatal parsing failures. All exception messages start with the following format:

[DocParser:<error code] details
Error Code Common causes

INVALID_ARGUMENT

Incorrect argument count, data type, or data form.

INVALID_OPTION

Unknown options, invalid option values, or a text encoding specified for a binary document.

FILE_NOT_FOUND

The file does not exist, is not a regular file, or the server process does not have permission to read it.

FILE_TOO_LARGE

The file or BLOB exceeds 100 MiB.

UNSUPPORTED_FORMAT

The ZIP/OOXML content is not a supported DOCX, PPTX, or XLSX.

CORRUPT_DOCUMENT

Corrupted signature, ZIP, XML, relationships, checksum, or document structure.

PASSWORD_REQUIRED

The PDF or OOXML document is encrypted; password input is not currently supported.

ENCODING_REQUIRED

The text has no Unicode BOM and is not valid UTF-8.

INVALID_ENCODING

The text bytes do not conform to the specified encoding or conflict with the BOM.

LIMIT_EXCEEDED

The page count, ZIP entry count, XML depth, output volume, or parsing time exceeds the limit.

OUT_OF_MEMORY

Memory allocation failure.

INTERNAL_ERROR

A runtime dependency such as PDFium behaves abnormally, or an unexpected parsing error occurs.