[FEATURE] Make import more robust by introducing timeout, recovery and watchdog #171

Merged

Dirk Koritnik merged 1 commit from bugfix/13000_import_reporting into main 2026-08-13 09:21:00 +00:00

Conversation 0 Commits 1 Files changed 62 +3300 -70

Anja Leichsenring commented 2026-08-12 15:17:07 +00:00

Member

Copy link

If upstream can't keep up with demand and stops answering requests,
timeouts for every request are imposed to avoid a stuck forever import run.
Additionally, retries are available so a failed request can recover.
The watchdog will keep an eye on the total runtime of an import and forcefully
terminate it, should it exceed its allocated budget. This abort will still enable
to persist collected data to database and write the log.
The log tells about those situations, should they occure, and so does the backend
module with a summary.

Extension configuration will provide values for the guard rails, with a hardcoded
fallback in place.

The existing FetchCache has been promoted from a runtime cache to a persitent one
with a livetime of currently 15 minutes, which will improve performance for imports
executed in short timely distance to each other. The livetime can be adjusted per
import within the import configuration.

Last but not least, import runs received output to give a sign of activity, the default
counting down processed root objects, and with -vvv a detailed list of retrieved URLs.

Resolves: https://redmine.werkraum-media.de/issues/13000
Assisted-by: Claude Opus 5 noreply@anthropic.com

If upstream can't keep up with demand and stops answering requests, timeouts for every request are imposed to avoid a stuck forever import run. Additionally, retries are available so a failed request can recover. The watchdog will keep an eye on the total runtime of an import and forcefully terminate it, should it exceed its allocated budget. This abort will still enable to persist collected data to database and write the log. The log tells about those situations, should they occure, and so does the backend module with a summary. Extension configuration will provide values for the guard rails, with a hardcoded fallback in place. The existing FetchCache has been promoted from a runtime cache to a persitent one with a livetime of currently 15 minutes, which will improve performance for imports executed in short timely distance to each other. The livetime can be adjusted per import within the import configuration. Last but not least, import runs received output to give a sign of activity, the default counting down processed root objects, and with `-vvv` a detailed list of retrieved URLs. Resolves: https://redmine.werkraum-media.de/issues/13000 Assisted-by: Claude Opus 5 noreply@anthropic.com

Anja Leichsenring added 1 commit 2026-08-12 15:17:07 +00:00

[FEATURE] Make import more robust by introducing timeout, recovery and watchdog ... 0c37dc6b47

If upstream can't keep up with demand and stops answering requests,
timeouts for every request are imposed to avoid a stuck forever import run.
Additionally, retries are available so a failed request can recover.
The watchdog will keep an eye on the total runtime of an import and forcefully
terminate it, should it exceed its allocated budget. This abort will still enable
to persist collected data to database and write the log.
The log tells about those situations, should they occure, and so does the backend
module with a summary.

Extension configuration will provide values for the guard rails, with a hardcoded
fallback in place.

The existing FetchCache has been promoted from a runtime cache to a persitent one
with a livetime of currently 15 minutes, which will improve performance for imports
executed in short timely distance to each other. The livetime can be adjusted per
import within the import configuration.

Last but not least, import runs received output to give a sign of activity, the default
counting down processed root objects, and with `-vvv` a detailed list of retrieved URLs.

Resolves: https://redmine.werkraum-media.de/issues/13000
Assisted-by: Claude Opus 5 noreply@anthropic.com

Anja Leichsenring force-pushed bugfix/13000_import_reporting from 0c37dc6b47 to 4cc9db0ef6 2026-08-13 08:49:28 +00:00 Compare

Dirk Koritnik merged commit 4cc9db0ef6 into main 2026-08-13 09:21:00 +00:00

Dirk Koritnik deleted branch bugfix/13000_import_reporting 2026-08-13 09:21:00 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

typo3/thuecat!171

No description provided.