KeyError Issue with HJDataset from LayoutParser #234

khanhthanhh9 · 2023-10-09T11:00:53Z

khanhthanhh9
Oct 9, 2023

I followed the "Running pre-trained models from Layout-Parser" tutorial in the DeepDoctection notebook and successfully used the Newspaper model from LayoutParser as shown in the second part of the tutorial. This model performed well in detecting various layouts in newspaper images.

@dd.object_types_registry.register("HJDatasetType")
class HJExtension(dd.ObjectTypes):
    """Additional HJDataset labels that were not previously registered"""
    pageframe = "Page Frame",
    row = "Row",
    titleregion = "Title Region",
    textregion = "Text Region",
    new_title = "New Title",
    subtitle = "Subtitle",
    otherstuff = "Other"

Then, I annotated the dataset as follows:

from deepdoctection.datapoint.view import IMAGE_ANNOTATION_TO_LAYOUTS, Layout
IMAGE_ANNOTATION_TO_LAYOUTS.update({i: Layout for i in HJExtension})

After this, I proceeded to add it to the layout service and perform recognition on the layout:

path_weights = dd.ModelCatalog.get_full_path_weights("/kaggle/temp/model_final.pth")
path_config = dd.ModelCatalog.get_full_path_configs("/kaggle/temp/model_final.pth")
categories = dd.ModelCatalog.get_profile("/kaggle/temp/model_final.pth").categories

d2_detector = dd.D2FrcnnDetector(path_config, path_weights, categories, config_overwrite=["NMS_THRESH_CLASS_AGNOSTIC=0.8", "MODEL.ROI_HEADS.SCORE_THRESH_TEST=0.1"])
image_layout = dd.ImageLayoutService(d2_detector)

page_parser = dd.PageParsingService(text_container=dd.LayoutType.word, floating_text_block_categories=[layout_item for layout_item in HJExtension])
pipe = dd.DoctectionPipe([image_layout], page_parsing_service=page_parser)

df = pipe.analyze(path="/kaggle/input/japanfileimage/file1")
df.reset_state()

df_iter = iter(df)
dp = next(df_iter)

image = dp.viz()

plt.figure(figsize = (25,17))
plt.axis('off')
plt.imshow(image)

However, there appears to be an issue with the category naming, which is causing the 'KeyError' for '7' in this context

`---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[8], line 6
      3 df.reset_state()
      5 df_iter = iter(df)
----> 6 dp = next(df_iter)
      8 image = dp.viz()
     10 plt.figure(figsize = (25,17))

File /opt/conda/lib/python3.10/site-packages/deepdoctection/dataflow/common.py:109, in MapData.__iter__(self)
    108 def __iter__(self) -> Iterator[Any]:
--> 109     for dp in self.df:
    110         ret = self.func(copy(dp))  # shallow copy the list
    111         if ret is not None:

File /opt/conda/lib/python3.10/site-packages/deepdoctection/dataflow/common.py:110, in MapData.__iter__(self)
    108 def __iter__(self) -> Iterator[Any]:
    109     for dp in self.df:
--> 110         ret = self.func(copy(dp))  # shallow copy the list
    111         if ret is not None:
    112             yield ret

File /opt/conda/lib/python3.10/site-packages/deepdoctection/pipe/base.py:93, in PipelineComponent.pass_datapoint(self, dp)
     91     with timed_operation(self.__class__.__name__):
     92         self.dp_manager.datapoint = dp
---> 93         self.serve(dp)
     94 else:
     95     self.dp_manager.datapoint = dp

File /opt/conda/lib/python3.10/site-packages/deepdoctection/pipe/layout.py:86, in ImageLayoutService.serve(self, dp)
     84 if self.padder:
     85     np_image = self.padder.apply_image(np_image)
---> 86 detect_result_list = self.predictor.predict(np_image)  # type: ignore
     87 if self.padder and detect_result_list:
     88     boxes = np.array([detect_result.box for detect_result in detect_result_list])

File /opt/conda/lib/python3.10/site-packages/deepdoctection/extern/d2detect.py:260, in D2FrcnnDetector.predict(self, np_img)
    248 """
    249 Prediction per image.
    250 
    251 :param np_img: image as numpy array
    252 :return: A list of DetectionResult
    253 """
    254 detection_results = d2_predict_image(
    255     np_img,
    256     self.d2_predictor,
    257     self.resizer,
    258     self.cfg.NMS_THRESH_CLASS_AGNOSTIC,
    259 )
--> 260 return self._map_category_names(detection_results)

File /opt/conda/lib/python3.10/site-packages/deepdoctection/extern/d2detect.py:271, in D2FrcnnDetector._map_category_names(self, detection_results)
    269 filtered_detection_result: List[DetectionResult] = []
    270 for result in detection_results:
--> 271     result.class_name = self._categories_d2[str(result.class_id)]
    272     if isinstance(result.class_id, int):
    273         result.class_id += 1

KeyError: '7'

You can check here for more info: Kaggle deepdoctection

The catalog of layout parser:
layoutparser catalog

Tutorial link:
Tutorial link

Answered by JaMe76

Oct 10, 2023

Yes, this is the stdout I was referring to. The model has a dense layer with 9 classes (8 + 1 background that will always be added).

So, it looks that there is one category missing.

As there is only this warning I doubt that there is an issue with the model architecture itself. There only seem to be one category missing.

View full answer

JaMe76 · 2023-10-09T21:44:52Z

JaMe76
Oct 9, 2023
Maintainer

While I cannot speak for the authors of Layoutparser, looking at the config file and the number of categories there is something inconsistent.

The config file for the model states that there are 8 classes, whereas they talk about HJDataset having only 7. Detectron2 assumes your classes having ids between 0 and num_classes -1. In deepdoctection we label between 1 and num_classes and decrease each value by 1 to be consistent with Detectron2.

I suggest to check the logs when the weights are being loaded. If the config is wrong the logs would stdout something like the top dense layer not being properly loaded.

3 replies

khanhthanhh9 Oct 10, 2023
Author

Thank you so much for the suggestion. I'm not sure why the config file states there are 8 classes! change in to 7 and it works now. Though the result might not be perfect or correct due to change

I have checked when the num_classes:8, the config does not logs anything about dense layer not being properly loaded.
However, there are warning info when i change the num_classes to 7:

path_weights = dd.ModelCatalog.get_full_path_weights("model/model_final_jap.pth")
path_config = dd.ModelCatalog.get_full_path_configs("model/model_final_jap.pth")
categories = dd.ModelCatalog.get_profile("model/model_final_jap.pth").categories

d2_detector = dd.D2FrcnnDetector(path_config,path_weights,categories,config_overwrite=["NMS_THRESH_CLASS_AGNOSTIC=0.8","MODEL.ROI_HEADS.SCORE_THRESH_TEST=0.1"])
image_layout = dd.ImageLayoutService(d2_detector)

page_parser = dd.PageParsingService(text_container = dd.LayoutType.word, # this argument is required but will not have any effect
floating_text_block_categories=[layout_item for layout_item in JapanExtension])
pipe = dd.DoctectionPipe([image_layout],page_parsing_service = page_parser)
[1010 09:24.11 @detection_checkpoint.py:38] INF [DetectionCheckpointer] Loading from /home/kimthanh1511/.cache/deepdoctection/weights/model/model_final_jap.pth ...
[1010 09:24.11 @checkpoint.py:150] INF [Checkpointer] Loading from /home/kimthanh1511/.cache/deepdoctection/weights/model/model_final_jap.pth ...
[1010 09:24.11 @checkpoint.py:338] WRN Skip loading parameter 'roi_heads.box_predictor.cls_score.weight' to the model due to incompatible shapes: (9, 1024) in the checkpoint but (8, 1024) in the model! You might want to double check if this is expected.
[1010 09:24.11 @checkpoint.py:338] WRN Skip loading parameter 'roi_heads.box_predictor.cls_score.bias' to the model due to incompatible shapes: (9,) in the checkpoint but (8,) in the model! You might want to double check if this is expected.
[1010 09:24.11 @checkpoint.py:338] WRN Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (32, 1024) in the checkpoint but (28, 1024) in the model! You might want to double check if this is expected.
[1010 09:24.11 @checkpoint.py:338] WRN Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (32,) in the checkpoint but (28,) in the model! You might want to double check if this is expected.
[1010 09:24.11 @checkpoint.py:350] WRN Some model parameters or buffers are not found in the checkpoint:
roi_heads.box_predictor.bbox_pred.{bias, weight}
roi_heads.box_predictor.cls_score.{bias, weight}

Do you have any idea how to fix these issues, seem like I might have to modify the config.yml futher?
Updated: : I tried HJ Dataset with the 'retinanet_R_50_FPN_3x' model and changed the config file to have 'num_classes' set to 7. However, I still encountered a 'KeyError' with the value '9'. Could this issue be related to dd library error?

JaMe76 Oct 10, 2023
Maintainer

Yes, this is the stdout I was referring to. The model has a dense layer with 9 classes (8 + 1 background that will always be added).

So, it looks that there is one category missing.

As there is only this warning I doubt that there is an issue with the model architecture itself. There only seem to be one category missing.

Answer selected by khanhthanhh9

khanhthanhh9 Oct 10, 2023
Author

This might be it, I looked into LayoutParser library issues and there are a lot of complaints about the model not specifying all the classes.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

KeyError Issue with HJDataset from LayoutParser #234

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

KeyError Issue with HJDataset from LayoutParser #234

Uh oh!

khanhthanhh9 Oct 9, 2023

Replies: 1 comment · 3 replies

Uh oh!

JaMe76 Oct 9, 2023 Maintainer

Uh oh!

Uh oh!

khanhthanhh9 Oct 10, 2023 Author

Uh oh!

Uh oh!

JaMe76 Oct 10, 2023 Maintainer

Uh oh!

khanhthanhh9 Oct 10, 2023 Author

khanhthanhh9
Oct 9, 2023

Replies: 1 comment 3 replies

JaMe76
Oct 9, 2023
Maintainer

khanhthanhh9 Oct 10, 2023
Author

JaMe76 Oct 10, 2023
Maintainer

khanhthanhh9 Oct 10, 2023
Author