* Enable graceful shutdown of rq workers
* Use `exec` in the `worker` command of the entrypoint to propagate
the `TERM` signal
* Allow rq processes managed by supervisor to exit without restart on
expected status codes
* Allow supervisorctl to contact the running supervisor
* Add a `shutdown_worker` command that will send `TERM` to all running
worker processes and then sleep. This allows orchestration systems
to initiate a graceful shutdown before sending `SIGTERM` to
supervisord
* Use Heroku worker as the BaseWorker
This implements a graceful shutdown on SIGTERM, which simplifies
external shutdown procedures.
* Fix imports based upon review
* Remove supervisorctl config
* enforce hard limits on non-responsive work horses by workers
* move differences from Worker to helper methods to help make the specialization clearer
* move HardLimitingWorker to redash/tasks
* move schedule.py to /tasks
* explain the motivation for HardLimitingWorker
* pleasing CodeClimate
* pleasing CodeClimate
* port query execution to RQ
* get rid of argsrepr
* avoid star imports
* allow queries to be cancelled in RQ
* return QueryExecutionErrors as job results
* fix TestTaskEnqueue and QueryExecutorTests
* remove Celery monitoring
* get rid of QueryTask and use RQ jobs directly (with a job serializer)
* Revert "remove Celery monitoring"
This reverts commit 37a74ea403301a21db9c36b3f05fbdc26cbd3530.
* reduce occurences of the word 'task'
* use Worker, Queue and Job instead of spreading names that share behavior details
* remove locks for failed jobs as well
* did I not commit that colon? oh my
* push the redis connection to RQ's stack on every request to avoid verbose connection setting
* use a connection context for tests
* remove Celery monitoring
* 👋 Celery
* remove Celery from Cypress
* black it up
* some more black
* return all started/queued job ids (for future monitoring
* Restyled by prettier (#4522)
* remove celery.py
* remove some frontend residuals that reappeared after a merge
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
* launch and monitor multiple workers using supervisor
* run supervisord in non-daemon mode
* redirect all output to stdout/stderr
* no need to log supervisord's output because it is redirected to stdout anyway
* updated and less brittle healthcheck
* add supervisor healthchecks
* remove redundant supervisor installation as it is installed by pip
* add a 5 minute check gate
* Make core app compatible with Python 3
No backward compatibility with Python 2.7 is kept.
This commit mostly contains changes made with 2to3 and manual
tweaking when necessary.
* Use Python 3.7 as base docker image
Since it is not possible to change redash/base:debian to Python 3
without breaking future relases, its Dockerfile is temporarly
copied here.
* Upgrade some requirements to newest versions
Some of the older versions were not compatible with Python 3.
* Migrate tests to Python 3
* Build frontend on Python 3
* Make the HMAC sign function compatible with Python 3
In Python 3, HMAC only works with bytes so the strings and the
float used in the sign function need to be encoded.
Hopefully this is still backward compatible with already generated
signatures.
* Use assertCountEqual instead of assertItemsEqual
The latter is not available in Python 3.
See https://bugs.python.org/issue17866
* Remove redundant encoding header for Python 3 modules
* Remove redundant string encoding in CLI
* Rename list() functions in CLI
These functions shadow the builtin list function which is
problematic since 2to3 adds a fair amount of calls to the builtin
list when it finds dict.keys() and dict.values().
Only the Python function is renamed, from the perspective of the
CLI nothing changes.
* Replace usage of Exception.message in CLI
`message` is not available anymore, instead use the string
representation of the exception.
* Adapt test handlers to Python 3
* Fix test that relied on dict ordering
* Make sure test results are always uploaded (#4215)
* Support encoding memoryview to JSON
psycopg2 returns `buffer` objects in Python 2.7 and `memoryview`
in Python 3. See #3156
* Fix test relying on object address ordering
* Decode bytes returned from Redis
* Stop using e.message for most exceptions
Exception.message is not available in Python 3 anymore, except
for some exceptions defined by third-party libraries.
* Fix writing XLSX files in Python 3
The buffer for the file should be made of bytes and the actual
content written to it strings.
Note: I do not know why the diff is so large as it's only a two
lines change. Probably a white space or file encoding issue.
* Fix test by comparing strings to strings
* Fix another exception message unavailable in Python 3
* Fix export to CSV in Python 3
The UnicodeWriter is not used anymore. In Python 3, the interface
provided by the CSV module only deals with strings, in and out.
The encoding of the output is left to the user, in our case
it is given to Flask via `make_response`.
* (Python 3) Use Redis' decode_responses=True option (#4232)
* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow)
* Make sure Redis connection uses decoded_responses option
* Remove unused imports.
* Use Redis' decode_responses option
* Remove cases of explicit Redis decoding
* Rename helper function and make sure it doesn't apply twice.
* Don't add decode_responses to Celery Redis connection URL
* Fix displaying error while connecting to SQLite
The exception message is always a string in Python 3, so no
need to try to decode things.
* Fix another missing exception message
* Handle JSON encoding for datasources returning bytes
SimpleJSON assumes the bytes it receives contain text data, so it
tries to UTF-8 encode them. It is sometimes not true, for instance
the SQLite datasource returns bytes for BLOB types, which typically
do not contain text but truly binary data.
This commit disables SimpleJSON auto encoding of bytes to str and
instead uses the same method as for memoryviews: generating a
hex representation of the data.
* Fix Python 3 compatibility with RQ
* Revert some changes 2to3 tends to do (#4261)
- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects.
- Also fixed some naming issues with one character variables in list comprehensions.
- Fix Flask warning.
* Upgrade dependencies
* Remove useless `iter` added by 2to3
* Fix get_next_path tests (#4280)
* Removed setting SERVER_NAME in tests setup to avoid a warning.
* Change get_next_path to not return empty string in case of a domain only value.
* Fix redirect tests:
Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path.
* Remove explicit dependency for Werkzeug
* Switched pytz and certifi to unbinded versions.
* Switch to new library for getting country from IP
`python-geoip-geolite2` is not compatible with Python 3, instead
use `maxminddb-geolite2` which is very similar as it includes
the geolite2 database in the package .
* Python 3 RQ modifications (#4281)
* show current worker job (alongside with minor cosmetic column tweaks)
* avoid loading entire job data for queued jobs
* track general RQ queues (default, periodic and schemas)
* get all active RQ queues
* call get_celery_queues in another place
* merge dicts the Python 3 way
* extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions
* Remove legacy Python flake8 tests
* add rq and an rq_worker service
* add rq_scheduler and an rq_scheduler service
* move beat schedule to periodic_jobs queue
* move version checks to RQ
* move query result cleanup to RQ
* use timedelta and DRY up a bit
* move custom tasks to RQ
* do actual schema refreshes in rq
* rename 'period_jobs' to 'periodic', as it obviously holds jobs
* move send_email to rq
* DRY up enqueues
* ditch and use a partially applied decorator
* move subscribe to rq
* move check_alerts_for_query to rq
* move record_event to rq
* make tests play nicely with rq
* 👋 beat
* rename rq_scheduler to plain scheduler, now that there's no Celery scheduler entrypoint
* add some color to rq-worker's output
* add logging context to rq jobs (while keeping execute_query context via get_task_logger for now)
* move schedule to its own module
* cancel previously scheduled periodic jobs. not sure this is a good idea.
* rename redash.scheduler to redash.schedule
* allow custom dynamic jobs to be added decleratively
* add basic monitoring to rq queues
* add worker monitoring
* pleasing the CodeClimate overlords
* adjust cypress docker-compose.yml to include rq changes
* DRY up Cypress docker-compose
* add rq dependencies to cypress docker-compose service
* an odd attempt at watching docker-compose logs when running with Cypress
* Revert "an odd attempt at watching docker-compose logs when running with Cypress"
This reverts commit 016bd1a93e3efa84a9f27d0f2acb972ce1957bcd.
* show docker-compose logs at Cypress shutdown
* Revert "DRY up Cypress docker-compose"
This reverts commit 43abac7084c207ab9e39192ac79d520448c2c527.
* minimal version for binding is 3.2
* remove unneccesary code reloads on cypress
* add a command which errors if any of the workers running inside the current machine haven't been active in the last minute
* SCHEMAS_REFRESH_QUEUE is no longer a required setting
* split tasks/queries.py to execution.py and maintenance.py
* fix tests after query execution split
* pleasing the CodeClimate overlords
* rename worker to celery_worker and rq_worker to worker
* use /rq_status instead of /jobs
* show started jobs' time ago according to UTC
* replace all spaces in column names
* fix query tests after execution split
* exit with an int
* general lint
* add an entrypoint for rq_healthcheck
* fix indentation
* delete all existing periodic jobs before scheduling them
* remove some unrequired requires
* move schedule example to redash.schedule
* add RQ integration to Sentry's setup
* pleasing the CodeClimate overlords
* remove replication settings from docker-compose - a proper way to scale using docker-compose would be the --scale CLI option, which will be described in the knowledge based
* revert to calling a function in dynamic settings to allow periodic jobs to be scheduled after app has been loaded
* don't need to depend on context when templating failure reports
* set the timeout_ttl to double the interval to avoid job results from expiring and having periodic jobs not reschedule
* whoops, bad merge
* describe custom jobs and don't actually schedule them
* fix merge
* Fix loading of periodic tasks and clean up extension loading.
This does a few things:
- add tests for extension loading
- refactor the extension and periodic task loading
- better handle assertions raised by extensions (e.g. when an extension tries to override an already registered view)
- attach exception traceback to error log during loading for improved debugging
* Use site.addsitedir instead of calling pip.
* Use sys.path instead of site.addsitedir and also the setup.py egg_info command.
* pick up *.py file changes and restart scheduler
* only watch /redash in order to avoid reloading on other file changes (e.g. tests)
* add dev_scheduler entrypoint
* use exec
* Update bin/docker-entrypoint
* rename dev_scheduler to dev_worker
* use same defaults as worker
* Decouple extensions from Flask app.
This separates the extension registry from the Flask app and also introduces a separate registry for preriodic tasks.
Fix#3466.
* Address review feedback.
* Update redash/extensions.py
Co-Authored-By: jezdez <jannis@leidel.info>
* Minor comment in requirements.
* Refactoring after getting feedback.
* Uncoupled bin/bundle-extensions from Flas app instance.
* Load bundles in bundle script and don’t rely on Flask.
* Upgraded to importlib-metadata 0.9.
* Add missing requirement.
* Fix TypeError.
* Added requirements for bundle_extension script.
* Install bundles requirement file correctly.
* Decouple bundle loading code from Redash.
* Install bundle requirements from requirements.txt.
* Use circleci/node for build-docker-image step, too.
* Fix Celery worker CLI parameter name that was changed in Celery 4.x.
* Set Celery worker --max-memory-per-child to 1/4th of total system memory.
* Review fixes.
* Review fixes.
* monitor "schemas" queue to run refresh_schema
`refresh_schema` tasks won't run because "schemas" queue isn't consumed
with default settings.
and it cause leaking redis storage, a "schemas" list is growing with time.
this PR fix it, adds "schemas" queue to monitor by celery.
* use scheduled_worker for "schemas" queue
instead of "adhoc_worker"
* open port 3000 for remote debugging
* add ptvsd
* use port 5678 to avoid changes in VSCode's default config
* attach to ptvsd
* no need to wait for attach
* actually, --debugger seems to be working
* create a new docker entry point for remote debugging
* alternative method to switch to debugging
By default Celery will use a file celerybeat-schedule in the current directory.
This is an issue in a Kubernetes/Openshift environment as the file may be lost or even impossible to write.
* Consistently use simplejson to loading and dumping JSON.
This introduces the new functions redash.utils.json_dumps and redash.utils.json_loads and simplifies the custom encoder setup.
UUIDs are now handled by the default encoder, too.
Fixes#2807.
Use string comparison in parse_boolean instead of the (simple)json module.
* Moved configuration to `.circleci/config.yml` as the documentation for V2 suggests.
* Created a dedicated Docker Compose configuration (in `.circleci/docker-compose.yml`) to remove volumes configuration as this is not supported with CircleCI's Docker executer.
* Fix the Docker image build and tarball packing jobs to work and use correct version.
Allow $REDASH_WEB_WORKERS to be set in the environment to change the
number of Gunicorn workers that are started (currently hardcoded to
four). If not set, the default is four, so this will not affect
existing users at all.
Documentated by example in docker-compose example manifests.
1. Simplify user creation to create a non system user (so the uid will usually
match the host user).
2. Set the user to redash & remove the need to change user in docker entrypoint.
