Skip to content

IMPR: 'import datajoint' slow on Mac amd64 #1220

Open
@drewyangdev

Description

@drewyangdev

Improvement Request

Problem

Not really a problem, but Mac import datajoint is way less performant comparing to import datajoint on Linux.

Requirements

# make a directory
mkdir test_dj_import
cd ~/Downloads/test_dj_import

## make import.py
echo "import datajoint" > import.py

## CLI
conda create -n dj313 -c defaults python=3.13
pip install datajoint snakeviz

# generate cProfiler result
python -m cProfile -o result.log import.py
# filter and sort result to top 20 longest cumulative cpu time - check cProfile doc 
python -c 'import pstats; p=pstats.Stats("result.log");p.sort_stats("cumulative").print_stats(20)' > sorted_result.log

# if you want more
# visualize non filtered result.log, since can't visualize filtered result, the func recursive format was broken
pip install snakeviz
snakeviz result.log
# you will get an error saying too many depth, just change the filter to "depth: 3", "cutoff: 1/100"

Justification

On either Mac/amd64 or Linux/amd64, the results are in the same order, but it takes ~6 sec to import on Mac, but ~1 sec on Linux.

On Mac/amd64

# sort_result.log
cat sorted_result.log | grep .py
        1    0.000    0.000    6.138    6.138 import.py:1(<module>)
        1    0.000    0.000    6.136    6.136 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/__init__.py:1(<module>)
        1    0.000    0.000    3.010    3.010 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/diagram.py:1(<module>)
        1    0.000    0.000    2.531    2.531 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/admin.py:1(<module>)
        1    0.000    0.000    1.986    1.986 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/connection.py:1(<module>)
        1    0.000    0.000    1.543    1.543 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/table.py:1(<module>)
        1    0.000    0.000    1.416    1.416 /Users/yam/opt/miniconda3/envs/dj313/lib/python3.13/site-packages/pandas/__init__.py:1(<module>)
        1    0.000    0.000    1.068    1.068 /Users/yam/opt/miniconda3/envs/dj313/lib/python3.13/site-packages/pandas/core/api.py:1(<module>)

On Linux/amd64

cat sorted_result.log | grep .py
        1    0.000    0.000    1.058    1.058 import.py:1(<module>)
        1    0.000    0.000    1.058    1.058 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/__init__.py:1(<module>)
        1    0.000    0.000    0.574    0.574 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/diagram.py:1(<module>)
        1    0.000    0.000    0.353    0.353 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/admin.py:1(<module>)
        1    0.000    0.000    0.310    0.310 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/connection.py:1(<module>)
        1    0.000    0.000    0.292    0.292 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/matplotlib/pyplot.py:1(<module>)
        1    0.000    0.000    0.203    0.203 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/table.py:1(<module>)
      106    0.002    0.000    0.185    0.002 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/matplotlib/artist.py:158(_update_set_signature_and_docstring)
      105    0.000    0.000    0.182    0.002 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/matplotlib/artist.py:125(__init_subclass__)
        1    0.000    0.000    0.169    0.169 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/pandas/__init__.py:1(<module>)

Please include steps to reproduce provided errors as follows:

  • OS (MACOS | Linux)
  • Python Version 3.13
  • DataJoint Version 0.14.3

Screenshots

Snakeviz on Mac
Image

Additional Research and Context

Metadata

Metadata

Labels

enhancementIndicates new improvementsstaleIndicates issues, pull requests, or discussions are inactive

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions