Enable Table Detection using table-transformer #249

vchow-allcloud · 2023-11-02T20:19:49Z

vchow-allcloud
Nov 2, 2023

I would like to do table detection and text extraction using the same analyzer. Using the default analyzer config, I am able to extract text effectively from PDFs. However, it appears tables are not turned on or detected by default.

I tried the following to enable table-transformers. I installed deepdoctection[pt], including detectron2. I also cloned the table transformers hugging face repo locally.

I then execute the following code:

config=["PT.LAYOUT.WEIGHTS=table-transformer-detection/pytorch_model.bin"]
analyzer = dd.get_dd_analyzer(config_overwrite=config)

but receive the error below.

[1102 12:54.11 @dd.py:391]  INF  Config: 
 {'DEVICE': 'cpu',
 'LANGUAGE': None,
 'LAYOUT_NMS_PAIRS': {'COMBINATIONS': None, 'PRIORITY': None, 'THRESHOLDS': None},
 'LIB': 'PT',
 'OCR': {'CONFIG': {'TESSERACT': 'dd/conf_tesseract.yaml'},
         'USE_DOCTR': False,
         'USE_TESSERACT': True,
         'USE_TEXTRACT': False,
         'WEIGHTS': {'DOCTR_RECOGNITION': {'PT': 'doctr/crnn_vgg16_bn/pt/crnn_vgg16_bn-9762b0b0.pt',
                                           'TF': 'doctr/crnn_vgg16_bn/tf/crnn_vgg16_bn-76b7f2c6.zip'},
                     'DOCTR_WORD': {'PT': 'doctr/db_resnet50/pt/db_resnet50-ac60cadc.pt',
                                    'TF': 'doctr/db_resnet50/tf/db_resnet50-adcafc63.zip'}}},
 'PT': {'CELL': {'FILTER': None,
                 'WEIGHTS': 'cell/d2_model_1849999_cell_inf_only.pt',
                 'WEIGHTS_TS': 'cell/d2_model_1849999_cell_inf_only.ts'},
        'ITEM': {'FILTER': None,
                 'PAD': {'BOTTOM': 60, 'LEFT': 60, 'RIGHT': 60, 'TOP': 60},
                 'WEIGHTS': 'item/d2_model_1639999_item_inf_only.pt',
                 'WEIGHTS_TS': 'item/d2_model_1639999_item_inf_only.ts'},
        'LAYOUT': {'FILTER': None,
                   'PAD': {'BOTTOM': 60, 'LEFT': 60, 'RIGHT': 60, 'TOP': 60},
                   'WEIGHTS': 'table-transformer-detection/pytorch_model.bin',
                   'WEIGHTS_TS': 'layout/d2_model_0829999_layout_inf_only.ts'}},
 'SEGMENTATION': {'ASSIGNMENT_RULE': 'ioa',
                  'CELL_CATEGORY_ID': 12,
                  'FULL_TABLE_TILING': True,
                  'REMOVE_IOU_THRESHOLD_COLS': 0.001,
                  'REMOVE_IOU_THRESHOLD_ROWS': 0.001,
                  'STRETCH_RULE': 'equal',
                  'THRESHOLD_COLS': 0.4,
                  'THRESHOLD_ROWS': 0.4},
 'TEXT_ORDERING': {'BROKEN_LINE_TOLERANCE': 0.003,
                   'FLOATING_TEXT_BLOCK_CATEGORIES': ['title', 'text', 'list'],
                   'HEIGHT_TOLERANCE': 2.0,
                   'INCLUDE_RESIDUAL_TEXT_CONTAINER': False,
                   'PARAGRAPH_BREAK': 0.035,
                   'STARTING_POINT_TOLERANCE': 0.005,
                   'TEXT_BLOCK_CATEGORIES': ['title', 'text', 'list', 'cell', 'column_header',
                                             'projected_row_header', 'spanning', 'row_header']},
 'TF': {'CELL': {'FILTER': None, 'WEIGHTS': 'cell/model-1800000_inf_only.data-00000-of-00001'},
        'ITEM': {'FILTER': None, 'WEIGHTS': 'item/model-1620000_inf_only.data-00000-of-00001'},
        'LAYOUT': {'FILTER': None, 'WEIGHTS': 'layout/model-800000_inf_only.data-00000-of-00001'}},
 'USE_LAYOUT': True,
 'USE_OCR': True,
 'USE_PDF_MINER': False,
 'USE_TABLE_REFINEMENT': True,
 'USE_TABLE_SEGMENTATION': True,
 'WORD_MATCHING': {'MAX_PARENT_ONLY': False,
                   'PARENTAL_CATEGORIES': ['text', 'title', 'list', 'cell', 'column_header',
                                           'projected_row_header', 'spanning', 'row_header'],
                   'RULE': 'ioa',
                   'THRESHOLD': 0.6}}
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[31], line 3
      1 config=["PT.LAYOUT.WEIGHTS=table-transformer-detection/pytorch_model.bin"]
----> 2 analyzer = dd.get_dd_analyzer(config_overwrite=config)

File ~/Documents/project/.venv/lib/python3.11/site-packages/deepdoctection/analyzer/dd.py:397, in get_dd_analyzer(reset_config_file, config_overwrite)
    394 if tensorpack_available():
    395     disable_tp_layer_logging()
--> 397 return build_analyzer(cfg)

File ~/Documents/project/.venv/lib/python3.11/site-packages/deepdoctection/analyzer/dd.py:241, in build_analyzer(cfg)
    238 pipe_component_list: List[PipelineComponent] = []
    240 if cfg.USE_LAYOUT:
--> 241     d_layout = _build_detector(cfg, "LAYOUT")
    242     layout = _build_service(d_layout, cfg, "LAYOUT")
    243     pipe_component_list.append(layout)

File ~/Documents/project/.venv/lib/python3.11/site-packages/deepdoctection/analyzer/dd.py:142, in _build_detector(cfg, mode)
    140 profile = ModelCatalog.get_profile(weights)
    141 categories = profile.categories
--> 142 assert categories is not None
    143 if profile.model_wrapper in ("TPFrcnnDetector",):
    144     return TPFrcnnDetector(config_path, weights_path, categories, filter_categories=filter_categories)

AssertionError:

Additional details:

Running this code on an M1 macbook pro
Python 3.11.5
deepdoctection[pt]==0.27

Thank you for your help!

JaMe76 · 2023-11-02T21:01:43Z

JaMe76
Nov 2, 2023
Maintainer

The reference to the model is not quite right: Use microsoft/table-transformer-detection/pytorch_model.bin instead of table-transformer-detection/pytorch_model.bin:

deepdoctection/deepdoctection/extern/model.py

Line 444 in 2a123fd

"microsoft/table-transformer-detection/pytorch_model.bin": ModelProfile(

Currently, there is no built-in method for rotating layout elements even if table transformers recognizes rotated tables.

2 replies

vchow-allcloud Nov 2, 2023
Author

Thank you! In the Get started tutorial, it looks like the analyzer is able to extract text and tables in one analysis. How do I do the same here? It looks like when I plug in the table-transformer model, text extraction is no longer completed. Thank you for your help!

JaMe76 Nov 2, 2023
Maintainer

The model does not behave like a layout model it only detects tables. Therefore, you do not have other layout segments and words outside of tables cannot be mapped to any other structure.

TEXT_ORDERING.INCLUDE_RESIDUAL_TEXT_CONTAINER=True will consider all words even if they do not have a match with a layout segment.

I suggest to look into this notebook. Table transformer might require some more configuration.

And as I said, it is currently not possible to process rotated tables correctly.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable Table Detection using table-transformer #249

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Enable Table Detection using table-transformer #249

Uh oh!

vchow-allcloud Nov 2, 2023

Replies: 1 comment · 2 replies

Uh oh!

JaMe76 Nov 2, 2023 Maintainer

Uh oh!

vchow-allcloud Nov 2, 2023 Author

Uh oh!

JaMe76 Nov 2, 2023 Maintainer

vchow-allcloud
Nov 2, 2023

Replies: 1 comment 2 replies

JaMe76
Nov 2, 2023
Maintainer

vchow-allcloud Nov 2, 2023
Author

JaMe76 Nov 2, 2023
Maintainer