In this article, we talk about the challenges and principles of extracting tabular data from PDF docs. We also compare six software tools to find out how they perform their respective tasks of parsing PDF tables and getting data out of them.

在本文中，我们讨论了从PDF文档中提取表格数据的挑战和原理。 我们还比较了六个软件工具，以了解它们如何执行各自的任务以解析PDF表并从中获取数据。

为什么从PDF中提取表格数据是一个挑战 (Why It’s a Challenge to Extract Tabular Data from PDF)

Today PDF is used as the basis of communication between companies, systems, and individuals. It is regarded as the standard for finalized versions of documents as it is not easily editable except in fillable PDF forms. Most popular use cases for PDF documents in the business environment are:

今天，PDF用作公司，系统和个人之间进行交流的基础。 它被认为是文档定稿版本的标准，因为除了可填写的PDF表格外，它不容易编辑。 在业务环境中，PDF文档的最流行用例是：

• Invoices
  发票
• Purchase Orders
  订单
• Shipping Notes
  运输注意事项
• Reports
  报告书
• Presentations
  简报
• Price & Product Lists
  价格和产品清单
• HR Forms
  人力资源表格

The sheer volume of information exchanged in PDF files means that the ability to extract data from PDF files quickly and automatically is essential. Spending time extracting data from PDFs to input into third party systems can be very costly for a company.

PDF文件中大量的信息交换意味着快速，自动地从PDF文件提取数据的能力至关重要。 花费时间从PDF提取数据以输入到第三方系统对于公司而言可能是非常昂贵的。

The main problem is that PDF was never really designed as a data input format, but rather, it was designed as an output format ensuring that data will look the same at any device and be printed correctly. A PDF file defines instructions to place characters (and other components) at precise x,y coordinates. Words are simulated by placing some characters closer than others. Spaces are simulated by placing words relatively far apart. As for tables — you are right — they are simulated by placing words as they would appear in a spreadsheet.

主要问题是PDF从来没有真正设计成数据输入格式，而是设计成输出格式，以确保数据在任何设备上看起来都一样并且可以正确打印。 PDF文件定义了将字符(和其他组件)放置在精确的x，y坐标处的指令。 通过将某些字符放置得比其他字符更近来模拟单词。 通过将单词相对分开放置来模拟空间。 至于表，您是对的，它们是通过像在电子表格中一样放置单词来模拟它们的。

We see that the PDF format has no internal representation of a table structure, which makes it difficult to extract tables for analysis. Unfortunately, a lot of open data is stored in PDFs, which was not designed for tabular data in the first place.

我们看到PDF格式没有表结构的内部表示，这使得提取表进行分析变得很困难。 不幸的是，许多开放数据存储在PDF中，而这些数据最初并不是为表格数据而设计的。

Luckily, different tools for extracting data from PDF tables are available in the market. Being somewhat similar to each other, they have their own advantages and disadvantages. In this article, we compare the most popular software that can help get tabular data out of PDFs and present it in an easy-to-read, editable, and searchable format.

幸运的是，市场上有多种用于从PDF表提取数据的工具。 它们彼此有点相似，但各有优缺点。 在本文中，我们将比较最流行的软件，该软件可以帮助从PDF中获取表格数据，并以易于阅读，可编辑和可搜索的格式显示它们。

OCR：何时以及为什么使用它 (OCR: When and Why to Use It)

Before choosing a tool, the first point is to understand what type of PDF files — text- or image-based — you will work with. It will impact on whether to use Optical Character Recognition (OCR) or not.

在选择工具之前，首先要了解您将使用哪种类型的PDF文件(基于文本或图像)。 这将影响是否使用光学字符识别(OCR)。

For example, we have a report generated as an output by a piece of software and imported in PDF format. Commonly, it is a text-based PDF.

例如，我们有一个报告，它是由一个软件生成的输出，并以PDF格式导入。 通常，它是基于文本的PDF。

If you work with image-based PDFs like scanned paper docs, or, for example, files captured by a digital camera — this is where OCR comes in. OCR enables machines to recognize written text or printed letters inside images. In other words, the technology helps to convert ‘text-as-an-image’ into an editable and searchable format.

如果您使用基于图像的PDF(例如，扫描的纸张文档)，或者使用数码相机捕获的文件，则可以使用OCR。OCR使机器可以识别图像中的文字或印刷字母。 换句话说，该技术有助于将“文本图像”转换为可编辑和可搜索的格式。

o if your PDF is image-based, then the process of data extraction consists of two tasks: to recognize text and then recognize the table structure (i.e., how the text is placed in rows and columns). Some tools, like Amazon Textract, can complete both of them. But since text recognition is a separate task, it can be performed independently, with the help of pure OCR tools. There are dozens of them, but in this article, we will focus on the table structure recognition.

o如果您的PDF是基于图像的，则数据提取过程包括两个任务：识别文本，然后识别表结构(即，文本在行和列中的放置方式)。 某些工具(例如Amazon Textract)可以完成这两个工具。 但由于文本识别是一项单独的任务，因此可以在纯OCR工具的帮助下独立执行。 它们有数十种，但是在本文中，我们将重点放在表结构识别上。

从表中检测和提取数据的细节 (Nuances of Detecting and Extracting Data from Tables)

Let’s assume that we have a text-based PDF document generated as an output by a piece of software. It contains tabular data, and we want to extract it and present in a digital format. There are two main ways to detect tables:

假设我们有一个基于文本的PDF文档，它是由一个软件生成的输出。 它包含表格数据，我们希望将其提取并以数字格式显示。 检测表有两种主要方法：

• Manually, when you detect column borders by eye and mark table columns by hands
  手动，当您用肉眼检测列边界并用手标记表格列时
• Automatically, when you rely on program algorithms
  自动，当您依赖程序算法时

Some tools offer either manual or automatic detection, while others combine both variants. From our experience, we can say that most of our clients’ cases require automatic recognition because they handle hundreds of documents with a slightly variable table structure. For example, columns’ width can differ not only between two documents but also on different pages inside one document. Therefore, if we mark each column in each table on each page by hand, it would take a lot of time and effort.

一些工具提供手动或自动检测，而其他工具则结合了这两种变体。 根据我们的经验，我们可以说大多数客户的案例需要自动识别，因为他们处理的表结构略有变化的数百个文档。 例如，列的宽度不仅可以在两个文档之间不同，而且可以在一个文档内的不同页面上不同。 因此，如果我们在每个页面上手动标记每个表中的每一列，则将花费大量时间和精力。

For our study, we created a sample one-page document covering all typical difficulties of data extraction caused by an ‘inconsistent’ table structure. Here is it:

对于我们的研究，我们创建了一个样本的单页文档，其中涵盖了由“不一致”表结构引起的所有典型数据提取困难。 就这个：

As you can see, in our sample document, we have multiple tables, and all of them have different widths of columns and inconsistent alignments. In such cases, manual detection can be tedious. For this reason, we are not going to use software that offers only manual table data detection. We choose an automatic way, and let’s see how good are algorithms for each of the tools.

如您所见，在我们的示例文档中，我们有多个表，并且所有表都有不同的列宽和不一致的对齐方式。 在这种情况下，手动检测可能很乏味。 因此，我们将不使用仅提供手动表格数据检测的软件。 我们选择一种自动方式，让我们看看每种工具的算法效果如何。

In addition to the table structure’s basic elements, we also deliberately included some ‘extra formatting’ that can potentially complicate the process of recognition:

除了表结构的基本元素外，我们还故意包含一些“额外格式”，这些格式可能会使识别过程复杂化：

• Multiple tables on a page — all of them should be detected
  页面上有多个表格-应该检测到所有这些表格
• Non-tabular data: headers and page number
  非表格数据：标题和页码
• The multiline text inside cells
  单元格内的多行文字
• Table column & row spanning — merged cells
  表的列和行跨度—合并的单元格
• Small table cell margins inside the table and between the table and the header
  表格内部以及表格和标题之间的小表格单元格边距

PDF表提取库和工具的比较 (Comparison of PDF Table Extraction Libraries and Tools)

From this study, you will learn about how six software tools perform their respective tasks of parsing PDF tables and how they stack up against each other. In the first part, we compare Tabula, PDFTron, and Amazon Textract.

通过本研究，您将了解六个软件工具如何执行各自的解析PDF表的任务，以及它们如何相互堆叠。 在第一部分中，我们比较了Tabula，PDFTron和Amazon Textract。

Let’s see how libraries and tools mentioned above coped with this task of data recognition and extraction based on our sample document.

让我们看看上面提到的库和工具如何根据我们的示例文档来完成数据识别和提取的任务。

塔布拉 (Tabula)

Tabula is a tool for liberating data tables locked inside PDF files. Tool overview:

Tabula是用于释放锁定在PDF文件中的数据表的工具。 工具概述：

• Type of software available: web application, requires simple local server setup

  可用软件类型： Web应用程序，需要简单的本地服务器设置

• Platforms: Windows, MacOS, open source (GitHub, MIT Licence)

  平台： Windows，MacOS，开源(GitHub，MIT Licence)

• Terms of use: free, open-source

  使用条款：免费，开源

• Supported output formats: CSV, TSV, JSON

  支持的输出格式： CSV，TSV，JSON

• Notes: Tabula only works on text-based PDFs, not scanned documents

  注意： Tabula仅适用于基于文本的PDF，不适用于扫描的文档

After uploading our sample file and parsing data from it via Tabula, we got the following output:

在上传示例文件并通过Tabula解析其中的数据之后，我们得到以下输出：

Zones marked in red are parts of the original file where Tabula detected tables. At this step, data recognition is captured correctly; all tables are detected, extraneous elements do not distort the result.

红色标记的区域是Tabula检测到表的原始文件的一部分。 在此步骤中，数据识别已正确捕获。 所有表都被检测到，无关元素不会扭曲结果。

But if we go further, we will see that the first row of the first and last rows of the last two tables is missing:

但是，如果走得更远，我们将看到最后两个表的第一行和最后一行的第一行丢失了：

Tabula offers two options: two data extraction methods. The automatic detection method is Lattice. The result of its usage you can see above. If we try Stream, results become better; all data is extracted correctly without missing rows:

Tabula提供了两种选择：两种数据提取方法。 自动检测方法是莱迪思。 您可以在上面看到其使用结果。 如果尝试Stream ，结果会更好； 正确提取所有数据而不会丢失行：

But using the Stream method, we face another problem: cells with multiline text are split into multiple table rows. It seems this variant is the best that we can get from Tabula.

但是使用Stream方法，我们面临另一个问题：具有多行文本的单元格被拆分为多个表行。 看来这种变形是我们可以从Tabula获得的最好的变形。

Summary: Tabula’s automatic detection method is not the best choice, so I don’t recommend relying on it. Both recognition methods provided by this tool have their own disadvantages. While data loss looks unacceptable in any case, split rows can be returned to their original state with additional processing — manually or with the script’s help.

简介 ：Tabula的自动检测方法不是最佳选择，因此我不建议依赖它。 此工具提供的两种识别方法都有其自身的缺点。 尽管在任何情况下数据丢失看起来都是不可接受的，但可以通过手动或在脚本的帮助下进行额外的处理，将拆分的行恢复为原始状态。

PDFTron (PDFTron)

PDFTron is software with multiple basic and advanced features that facilitate the manipulation of PDFs. Tool overview:

PDFTron是具有多种基本和高级功能的软件，可简化对PDF的操作。 工具概述：

• Type of software available: desktop app, web browser app, mobile app
  可用软件类型：桌面应用程序，Web浏览器应用程序，移动应用程序
• Platforms: Android, iOS, Windows, Linux, MacOS
  平台：Android，iOS，Windows，Linux，MacOS
• Terms of use: free trial period, pricing for licensing starts at $4000 annually
  使用条款：免费试用期，许可的起价为每年4000美元
• Supported output formats: MS Word, SVG, HTML
  支持的输出格式：MS Word，SVG，HTML

After uploading our sample file and parsing data from it via PDFTron, we got the following output:

在上传示例文件并通过PDFTron解析了其中的数据之后，我们得到以下输出：

Red rectangles show the borders of detected tables. The number of tables and tabular data is recognized correctly, but headers of the third and fourth tables are captured as the table elements.

红色矩形显示检测到的表格的边框。 可以正确识别表和表格数据的数量，但是将第三和第四表的标题捕获为表元素。

If we convert the original PDF into HTML format using PDFTron, we’ll see that the headers named ‘CATEGORY 2’ and ‘CATEGORY 3’ are included as separate cells inside the table. Also, there are bugs with merged cells of the first table: the last two columns are merged as a larger one, and the second line of text is separated into a separate cell.

如果使用PDFTron将原始PDF转换为HTML格式，则会看到名为“ CATEGORY 2”和“ CATEGORY 3”的标头作为单独的单元格包含在表中。 此外，第一个表的合并单元格也存在错误：最后两列合并为较大的单元格，第二行文本被分隔为单独的单元格。

Summary: In the output document we can see a piece of non-tabular data incorrectly included in the extraction result. It’s not a big problem since it can be purged after processing, manually, or with the script’s help. Also, I will not recommend this tool in cases when you have a lot of merged cells — they might be messed up.

简介：在输出文档中，我们可以看到提取结果中错误地包含了一段非表格数据。 这不是一个大问题，因为可以在处理后，手动或在脚本的帮助下清除它。 另外，如果您有很多合并的单元格，则我不推荐使用此工具，因为它们可能会弄糟。

亚马逊Textract (Amazon Textract)

Amazon Textract is a service that automatically extracts text and data from scanned documents that go beyond simple optical character recognition (OCR) to identify, understand, and extract data from forms and tables. Tool overview:

Amazon Textract是一项服务，可以自动从扫描的文档中提取文本和数据，而不仅仅是简单的光学字符识别(OCR)来识别，理解和提取表单和表格中的数据。 工具概述：

• Type of software available: web application
  可用软件类型：Web应用程序
• Platforms: any modern web browser, all processing goes ‘in the Cloud’
  平台：任何现代的Web浏览器，所有处理都在“云端”进行
• Terms of use: several models (free, monthly subscriptions, cost-per-page model)
  使用条款：多种模式(免费，每月订阅，每页成本模式)
• Supported output formats: raw JSON, JSON for each page in the document, text, key/values exported as CSV, tables exported as CSV.
  支持的输出格式：原始JSON，文档中每个页面的JSON，文本，以CSV格式导出的键/值，以CSV格式导出的表。

After uploading our sample file and parsing data from it via Amazon Textract, we got the following result:

在上传示例文件并通过Amazon Textract解析了其中的数据之后，我们得到了以下结果：

Zones painted in gray are parts of the original file where Amazon Textract detected tables. As you can see, it didn’t recognize the first table on the page, detected two separate tables — the second and the third — as a single one, and also messed up the order of tables on the page.

灰色区域是Amazon Textract检测到表的原始文件的一部分。 如您所见，它无法识别页面上的第一个表，将两个单独的表(第二个和第三个表)检测为单个表，并且还弄乱了页面上表的顺序。

Regarding the data inside the cells, the extraction result is quite satisfying with some exceptions:

关于单元内的数据，提取结果非常令人满意，但有一些例外：

• Missing the data of the first table in the extraction result
  提取结果中缺少第一个表的数据
• Header string ‘CATEGORY 3 is included as a part of table in the extraction result
  标头字符串“ CATEGORY 3”作为表的一部分包含在提取结果中

Summary: Amazon Textract looks to be the less suitable tool for data extraction applied to this particular case. A large amount of data loss, messed order of tables, and non-tabular data in the output document. The main advantage of Amazon Textract compared to all other tools is OCR, so that you can extract tabular data from images or image-based PDFs. In the case of text-based PDF, I would recommend choosing another tool.

简介： Amazon Textract似乎不太适合应用于此特定案例的数据提取工具。 输出文档中大量数据丢失，表顺序混乱以及非表格数据。 与所有其他工具相比，Amazon Textract的主要优势是OCR，因此您可以从图像或基于图像的PDF中提取表格数据。 对于基于文本的PDF，我建议选择其他工具。

Here is the first part of the article ‘How to Extract Tabular Data from PDF. In the second part, which is coming soon, we will analyze three more popular solutions for extracting and converting data from PDF and prepare a big сomparison table where each tool is rated according to the specific criteria. Follow UpsilonIT’s blog not to miss the great content!

这是文章“如何从PDF提取表格数据”的第一部分。 在 即将推出 的第二部分中 ，我们将分析另外三种从PDF提取和转换数据的流行解决方案，并准备 一张巨大的比较表 ，其中每个工具均根据特定标准进行评级。 关注UpsilonIT的博客，不要错过精彩的内容！