Tests: Adjust software tests for CrateDB #284
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amotl
force-pushed
the
cratedb-tests
branch
3 times, most recently
from
24d5131 to
8466d1c
Compare
amotl
commented
Jul 6, 2025
Comment on lines
+819
to
+832
| # CrateDB needs an explicit flush to make data available for reads immediately. | ||
| if dest_engine.dialect.name == "crate": | ||
| dest_engine.execute(f"REFRESH TABLE {schema_rand_prefix}.output") |
There was a problem hiding this comment.
Note to self: Is it really needed? Doesn't the dlt adapter already flush the relevant table after concluding the load operation, or is it missing in ingestr?
Comment on lines
-710
to
+846
| assert res[0] == (1, "val1", as_datetime("2022-01-01")) | ||
| assert res[1] == (2, "val2", as_datetime("2022-02-01")) | ||
|
|
||
| # Compensate for CrateDB types and insert order. | ||
| if dest_connection_url.startswith("cratedb://"): | ||
| assert (1, "val1", 1640995200000) in res | ||
| assert (2, "val2", 1643673600000) in res | ||
| else: | ||
| assert res[0] == (1, "val1", as_datetime("2022-01-01")) | ||
| assert res[1] == (2, "val2", as_datetime("2022-02-01")) |
There was a problem hiding this comment.
Well. C'est la vie. Let me add a few thoughts about it.
Maybe improve this spot by patching as_datetime() at runtime in the context of CrateDB instead, in order to reduce the cluttering on the actual data validation?
On the other hand, aren't there corresponding converter hooks in crate-python already, which could make it work fluently?
Is it even a concern of improving the SQLAlchemy dialect with relevant features?
Comment on lines
+849
to
+870
| # CrateDB: Compensate for "Type `date` does not support storage". | ||
| updated_at_type = "DATE" | ||
| if dest_connection_url.startswith("cratedb://"): | ||
| updated_at_type = "TIMESTAMP" | ||
|
|
||
| source_engine = sqlalchemy.create_engine(source_connection_url) | ||
| with source_engine.begin() as conn: | ||
| conn.execute(f"DROP SCHEMA IF EXISTS {schema_rand_prefix}") | ||
| conn.execute(f"CREATE SCHEMA {schema_rand_prefix}") | ||
| conn.execute( | ||
| f"CREATE TABLE {schema_rand_prefix}.input (id INTEGER, val VARCHAR(20), updated_at DATE)" | ||
| f"CREATE TABLE {schema_rand_prefix}.input (id INTEGER, val VARCHAR(20), updated_at {updated_at_type})" |
There was a problem hiding this comment.
amotl
force-pushed
the
cratedb-tests
branch
3 times, most recently
from
e1c80a6 to
4cd8559
Compare
amotl
changed the title
amotl
force-pushed
the
cratedb-tests
branch
6 times, most recently
from
45f2c7c to
d20b837
Compare
amotl
commented
Jul 6, 2025
Makefile
Outdated
Comment on lines
25
to
28
| set -a; source test.env; set +a; TESTCONTAINERS_RYUK_DISABLED=true pytest -n auto -x -rP -vv --tb=short --durations=10 --cov=ingestr --no-cov-on-fail | ||
| set -a; source test.env; set +a; TESTCONTAINERS_RYUK_DISABLED=true pytest -n auto -x -rP -vv --tb=short --durations=10 --cov=ingestr --no-cov-on-fail -k "not cratedb" | ||
| set -a; source test.env; set +a; TESTCONTAINERS_RYUK_DISABLED=true pytest -rP -vv --tb=short --durations=10 --cov=ingestr --no-cov-on-fail -k "cratedb" |
There was a problem hiding this comment.
We discovered that the CrateDB Testcontainers implementation is not ready to be used together with xdist yet, so we are invoking integration tests against CrateDB separately now.
amotl
force-pushed
the
cratedb-tests
branch
2 times, most recently
from
db792c9 to
0fc85de
Compare
This was referenced Jul 6, 2025
Closed
... so invoke software tests on GHA separately.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About
Retroactively add software tests for CrateDB by adjusting the existing test suite.
It is very unlikely the patch will get merged in its current form, so it will need to be maintained separately to provide quality assurance for the CrateDB source and destination adapters.
Thoughts
The patch clearly demonstrates where CrateDB is a different animal compared to other DBMS. We don't know yet how to compress or re-shape those anomaly-balancing boilerplate adjustments, in order to reduce the noise of the patch, but we hope to be able to resolve one or another obstacle for the better as we go.
Backlog
main_test.py, expand intosources_test.pyanddestinations_test.pyas well.cratedb-toolkit.References