redash/tests/handlers/test_query_results.py

331 lines
14 KiB
Python
Raw Normal View History

2015-10-16 20:11:19 +00:00
from tests import BaseTestCase
2016-11-30 14:52:30 +00:00
from redash.models import db
from redash.utils import json_dumps
from redash.handlers.query_results import error_messages
2015-10-16 20:11:19 +00:00
class TestQueryResultsCacheHeaders(BaseTestCase):
def test_uses_cache_headers_for_specific_result(self):
query_result = self.factory.create_query_result()
query = self.factory.create_query(latest_query_data=query_result)
rv = self.make_request('get', '/api/queries/{}/results/{}.json'.format(query.id, query_result.id))
self.assertIn('Cache-Control', rv.headers)
2015-10-16 20:11:19 +00:00
def test_doesnt_use_cache_headers_for_non_specific_result(self):
query_result = self.factory.create_query_result()
query = self.factory.create_query(latest_query_data=query_result)
rv = self.make_request('get', '/api/queries/{}/results.json'.format(query.id))
self.assertNotIn('Cache-Control', rv.headers)
2015-10-16 20:11:19 +00:00
def test_returns_404_if_no_cached_result_found(self):
query = self.factory.create_query(latest_query_data=None)
rv = self.make_request('get', '/api/queries/{}/results.json'.format(query.id))
self.assertEqual(404, rv.status_code)
class TestQueryResultListAPI(BaseTestCase):
def test_get_existing_result(self):
query_result = self.factory.create_query_result()
query = self.factory.create_query()
rv = self.make_request('post', '/api/query_results',
data={'data_source_id': self.factory.data_source.id,
2016-11-30 14:52:30 +00:00
'query': query.query_text})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
self.assertEqual(query_result.id, rv.json['query_result']['id'])
def test_execute_new_query(self):
query_result = self.factory.create_query_result()
query = self.factory.create_query()
rv = self.make_request('post', '/api/query_results',
data={'data_source_id': self.factory.data_source.id,
2016-11-30 14:52:30 +00:00
'query': query.query_text,
'max_age': 0})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
self.assertNotIn('query_result', rv.json)
self.assertIn('job', rv.json)
def test_execute_query_without_access(self):
2016-11-30 14:52:30 +00:00
group = self.factory.create_group()
db.session.commit()
user = self.factory.create_user(group_ids=[group.id])
query = self.factory.create_query()
rv = self.make_request('post', '/api/query_results',
data={'data_source_id': self.factory.data_source.id,
2016-11-30 14:52:30 +00:00
'query': query.query_text,
'max_age': 0},
user=user)
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
self.assertIn('job', rv.json)
def test_execute_query_with_params(self):
query = "SELECT {{param}}"
rv = self.make_request('post', '/api/query_results',
data={'data_source_id': self.factory.data_source.id,
'query': query,
'max_age': 0})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 400)
self.assertIn('job', rv.json)
rv = self.make_request('post', '/api/query_results',
data={'data_source_id': self.factory.data_source.id,
'query': query,
'parameters': {'param': 1},
'max_age': 0})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
self.assertIn('job', rv.json)
rv = self.make_request('post', '/api/query_results?p_param=1',
data={'data_source_id': self.factory.data_source.id,
'query': query,
'max_age': 0})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
self.assertIn('job', rv.json)
2016-05-30 15:30:05 +00:00
def test_execute_on_paused_data_source(self):
self.factory.data_source.pause()
rv = self.make_request('post', '/api/query_results',
data={'data_source_id': self.factory.data_source.id,
'query': 'SELECT 1',
'max_age': 0})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 400)
2016-05-30 15:30:05 +00:00
self.assertNotIn('query_result', rv.json)
self.assertIn('job', rv.json)
Allow Parameters on Public Dashboards (#3659) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * enqueue jobs for ApiUsers * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * remove residues from bad rebase * add query id and data source id to serialized public dashboards * add global parameters directive to public dashboards page * allow access to a query by the api_key of the dashboard which includes it * rename `object` to `obj` * simplify permission tests once `has_access` accepts groups * support global parameters for public dashboards * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * simplify permission tests once `has_access` accepts groups * no need to log `is_api_key` * send parameters to public dashboard page * allow access to a query by the api_key of the dashboard which includes it * disable sharing if dashboard is associated with unsafe queries * remove cypress test added in the wrong place due to a faulty rebase * add support for clicking buttons in cy.clickThrough * Cypress test which verifies that dashboards with safe queries can be shared * Cypress test which verifies that dashboards with unsafe queries can't be shared * remove duplicate tests * use this.enabled and negate when needed * remove stale comment * add another Cypress test to verify that unauthenticated users have access to public dashboards with parameters * obviously, I commit 'only' the first time I use it * search for query access by query id and not api_key * no need to fetch latest query data as it is loaded by frontend from the textless endpoint * test that queries associated with dashboards are accessible when supplying the dashboard api_key * propagate `isDirty` down to `QueryBasedParameterInput` * go to /api/:id/dropdown while editing a query, since dropdown queries might still not be associated with the parent. see #3711 * show helpful error message if dropdown values cannot be fetched * use backticks instead of line concatenation * remove requirement to have direct access to dropdown query in order validate it. parent query association checks are sufficient * remove isDirty-based implementation and allow dropdown queries through nested ACL even if they aren't associated yet (given that the user has _direct_ access to the dropdown query) * fix tests to cover all cases for /api/queries/:id/dropdowns/:id * fix indentation * require access to the query, not the data source * resolve dashboard user by query id * apply new copy to Cypress tests * if only something would have prevented me from commiting an 'only' call 🤔 * very important handling of whitespace * respond to parameter's Apply button * text widgets are safe for sharing * remove redundant event * add a safety check that object has dashboard_api_keys before calling it * supply a parameter value for text parameters to have it show up * add parameter values for date and datetime * use the current year and month to avoid pagination * use Cypress.moment() instead of preinstalled moment() * explicitly create parameters * refresh query data if a querystring parameter is provided * avoid sending a data_source_id - it's only relevant to unsaved queries, since a saved query's data_source is available in the backend * remove empty query text workaround * provide default value to parameter * add a few more dashboard sharing specs * lint * wait for DynamicTable to appear to reveal that actual results are displaying * override error message for unsafely shared widgets
2019-07-15 12:09:30 +00:00
def test_execute_without_data_source(self):
rv = self.make_request('post', '/api/query_results',
data={'query': 'SELECT 1',
'max_age': 0})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 401)
self.assertDictEqual(rv.json, error_messages['select_data_source'][0])
Allow Parameters on Public Dashboards (#3659) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * enqueue jobs for ApiUsers * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * remove residues from bad rebase * add query id and data source id to serialized public dashboards * add global parameters directive to public dashboards page * allow access to a query by the api_key of the dashboard which includes it * rename `object` to `obj` * simplify permission tests once `has_access` accepts groups * support global parameters for public dashboards * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * simplify permission tests once `has_access` accepts groups * no need to log `is_api_key` * send parameters to public dashboard page * allow access to a query by the api_key of the dashboard which includes it * disable sharing if dashboard is associated with unsafe queries * remove cypress test added in the wrong place due to a faulty rebase * add support for clicking buttons in cy.clickThrough * Cypress test which verifies that dashboards with safe queries can be shared * Cypress test which verifies that dashboards with unsafe queries can't be shared * remove duplicate tests * use this.enabled and negate when needed * remove stale comment * add another Cypress test to verify that unauthenticated users have access to public dashboards with parameters * obviously, I commit 'only' the first time I use it * search for query access by query id and not api_key * no need to fetch latest query data as it is loaded by frontend from the textless endpoint * test that queries associated with dashboards are accessible when supplying the dashboard api_key * propagate `isDirty` down to `QueryBasedParameterInput` * go to /api/:id/dropdown while editing a query, since dropdown queries might still not be associated with the parent. see #3711 * show helpful error message if dropdown values cannot be fetched * use backticks instead of line concatenation * remove requirement to have direct access to dropdown query in order validate it. parent query association checks are sufficient * remove isDirty-based implementation and allow dropdown queries through nested ACL even if they aren't associated yet (given that the user has _direct_ access to the dropdown query) * fix tests to cover all cases for /api/queries/:id/dropdowns/:id * fix indentation * require access to the query, not the data source * resolve dashboard user by query id * apply new copy to Cypress tests * if only something would have prevented me from commiting an 'only' call 🤔 * very important handling of whitespace * respond to parameter's Apply button * text widgets are safe for sharing * remove redundant event * add a safety check that object has dashboard_api_keys before calling it * supply a parameter value for text parameters to have it show up * add parameter values for date and datetime * use the current year and month to avoid pagination * use Cypress.moment() instead of preinstalled moment() * explicitly create parameters * refresh query data if a querystring parameter is provided * avoid sending a data_source_id - it's only relevant to unsaved queries, since a saved query's data_source is available in the backend * remove empty query text workaround * provide default value to parameter * add a few more dashboard sharing specs * lint * wait for DynamicTable to appear to reveal that actual results are displaying * override error message for unsafely shared widgets
2019-07-15 12:09:30 +00:00
class TestQueryResultAPI(BaseTestCase):
def test_has_no_access_to_data_source(self):
ds = self.factory.create_data_source(group=self.factory.create_group())
query_result = self.factory.create_query_result(data_source=ds)
rv = self.make_request('get', '/api/query_results/{}'.format(query_result.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
def test_has_view_only_access_to_data_source(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=True)
query_result = self.factory.create_query_result(data_source=ds)
rv = self.make_request('get', '/api/query_results/{}'.format(query_result.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
def test_has_full_access_to_data_source(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=False)
query_result = self.factory.create_query_result(data_source=ds)
rv = self.make_request('get', '/api/query_results/{}'.format(query_result.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
def test_execute_new_query(self):
query = self.factory.create_query()
rv = self.make_request('post', '/api/queries/{}/results'.format(query.id), data={'parameters': {}})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
self.assertIn('job', rv.json)
def test_execute_but_has_no_access_to_data_source(self):
ds = self.factory.create_data_source(group=self.factory.create_group())
query = self.factory.create_query(data_source=ds)
rv = self.make_request('post', '/api/queries/{}/results'.format(query.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
self.assertDictEqual(rv.json, error_messages['no_permission'][0])
def test_execute_with_no_parameter_values(self):
query = self.factory.create_query()
rv = self.make_request('post', '/api/queries/{}/results'.format(query.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
self.assertIn('job', rv.json)
Be more permissive when parameters are safe (#3383) * use the textless endpoint (/api/queries/:id/results) for pristine queriest * reverse conditional. not not is making me the headaches. * add ParameterizedQuery#is_safe with an inital naive implementation which treats any query with a text parameter as not safe. This will be remedied later when DB drivers will handle these parameters. * allow getting new query results even if user has only view permissions to the data source (given that the query is safe) * fix lint error - getDerivedStateFromProps should be placed after state * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * move execution preparation to a different function, which will be soon reused * go to textless /api/queries/:id/results by default * let the query view decide if text or textless endpoint is needed * allow safe queries to be executed in the UI even if the user has no permission to execute and create new query results * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * use dict#get instead of a None guard * use ParameterizedQuery in queries handler as well * test that /queries/:id/results allows execution of safe queries even if user has view_only permissions * lint * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * remove unused methods * avoid cyclic imports by importing only when needed * verify that a ParameterizedQuery without any parameters is considered safe * introduce query.parameter_schema * encapsulate ParameterizedQuery creation inside Query
2019-02-26 18:55:01 +00:00
def test_prevents_execution_of_unsafe_queries_on_view_only_data_sources(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=True)
query = self.factory.create_query(data_source=ds, options={"parameters": [{"name": "foo", "type": "text"}]})
rv = self.make_request('post', '/api/queries/{}/results'.format(query.id), data={"parameters": {}})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
self.assertDictEqual(rv.json, error_messages['unsafe_on_view_only'][0])
Be more permissive when parameters are safe (#3383) * use the textless endpoint (/api/queries/:id/results) for pristine queriest * reverse conditional. not not is making me the headaches. * add ParameterizedQuery#is_safe with an inital naive implementation which treats any query with a text parameter as not safe. This will be remedied later when DB drivers will handle these parameters. * allow getting new query results even if user has only view permissions to the data source (given that the query is safe) * fix lint error - getDerivedStateFromProps should be placed after state * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * move execution preparation to a different function, which will be soon reused * go to textless /api/queries/:id/results by default * let the query view decide if text or textless endpoint is needed * allow safe queries to be executed in the UI even if the user has no permission to execute and create new query results * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * use dict#get instead of a None guard * use ParameterizedQuery in queries handler as well * test that /queries/:id/results allows execution of safe queries even if user has view_only permissions * lint * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * remove unused methods * avoid cyclic imports by importing only when needed * verify that a ParameterizedQuery without any parameters is considered safe * introduce query.parameter_schema * encapsulate ParameterizedQuery creation inside Query
2019-02-26 18:55:01 +00:00
def test_allows_execution_of_safe_queries_on_view_only_data_sources(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=True)
query = self.factory.create_query(data_source=ds, options={"parameters": [{"name": "foo", "type": "number"}]})
rv = self.make_request('post', '/api/queries/{}/results'.format(query.id), data={"parameters": {}})
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
Be more permissive when parameters are safe (#3383) * use the textless endpoint (/api/queries/:id/results) for pristine queriest * reverse conditional. not not is making me the headaches. * add ParameterizedQuery#is_safe with an inital naive implementation which treats any query with a text parameter as not safe. This will be remedied later when DB drivers will handle these parameters. * allow getting new query results even if user has only view permissions to the data source (given that the query is safe) * fix lint error - getDerivedStateFromProps should be placed after state * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * move execution preparation to a different function, which will be soon reused * go to textless /api/queries/:id/results by default * let the query view decide if text or textless endpoint is needed * allow safe queries to be executed in the UI even if the user has no permission to execute and create new query results * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * use dict#get instead of a None guard * use ParameterizedQuery in queries handler as well * test that /queries/:id/results allows execution of safe queries even if user has view_only permissions * lint * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * remove unused methods * avoid cyclic imports by importing only when needed * verify that a ParameterizedQuery without any parameters is considered safe * introduce query.parameter_schema * encapsulate ParameterizedQuery creation inside Query
2019-02-26 18:55:01 +00:00
Sharing embeds with safe parameters (#3495) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * support querystring params * extract coercing of numbers to function, along with a friendlier implementation * wire embeds to textless endpoint * allow users with view_only permissions to execute queries on the textless endpoint, as it only allows safe queries to run * enqueue jobs for ApiUsers * add parameters component for embeds * include existing parameters in embed code * fetch correct values for json requests * remove previous embed parameter code * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up) * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message on setting deprecation * rephrase deprecation message * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * change has_access and require_access signatures to work with the objects that require access, instead of their groups * split has_access between normal users and ApiKey users * remove residues from bad rebase * allow access to safe queries via api keys * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * fix bad rebase * send embed parameters through POST data * no need to log `is_api_key` * move query fetching by api_key to within the Query model * fetch user by adding a get_by_id function on the User model * pass parameters as POST data (fixes test failure introduced by switching from query string parameters to POST data) * test the right thing - queries with safe parameters should be embeddable * introduce cy.clickThrough * add another Cypress test to make sure unsafe queries cannot be embedded * serialize Parameters into query string * set is_api_key as the last parameter to (hopefully) avoid backward-dependency problems * Update redash/models/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * attempt to fix empty percy snapshots * snap percies after DOM is fully loaded
2019-04-02 08:45:38 +00:00
def test_prevents_execution_of_unsafe_queries_using_api_key(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=True)
query = self.factory.create_query(data_source=ds, options={"parameters": [{"name": "foo", "type": "text"}]})
data = {'parameters': {'foo': 'bar'}}
rv = self.make_request('post', '/api/queries/{}/results?api_key={}'.format(query.id, query.api_key), data=data)
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
self.assertDictEqual(rv.json, error_messages['unsafe_when_shared'][0])
Sharing embeds with safe parameters (#3495) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * support querystring params * extract coercing of numbers to function, along with a friendlier implementation * wire embeds to textless endpoint * allow users with view_only permissions to execute queries on the textless endpoint, as it only allows safe queries to run * enqueue jobs for ApiUsers * add parameters component for embeds * include existing parameters in embed code * fetch correct values for json requests * remove previous embed parameter code * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up) * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message on setting deprecation * rephrase deprecation message * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * change has_access and require_access signatures to work with the objects that require access, instead of their groups * split has_access between normal users and ApiKey users * remove residues from bad rebase * allow access to safe queries via api keys * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * fix bad rebase * send embed parameters through POST data * no need to log `is_api_key` * move query fetching by api_key to within the Query model * fetch user by adding a get_by_id function on the User model * pass parameters as POST data (fixes test failure introduced by switching from query string parameters to POST data) * test the right thing - queries with safe parameters should be embeddable * introduce cy.clickThrough * add another Cypress test to make sure unsafe queries cannot be embedded * serialize Parameters into query string * set is_api_key as the last parameter to (hopefully) avoid backward-dependency problems * Update redash/models/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * attempt to fix empty percy snapshots * snap percies after DOM is fully loaded
2019-04-02 08:45:38 +00:00
def test_access_with_query_api_key(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=False)
query = self.factory.create_query()
query_result = self.factory.create_query_result(data_source=ds, query_text=query.query_text)
rv = self.make_request('get', '/api/queries/{}/results/{}.json?api_key={}'.format(query.id, query_result.id, query.api_key), user=False)
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
def test_access_with_query_api_key_without_query_result_id(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=False)
query = self.factory.create_query()
query_result = self.factory.create_query_result(data_source=ds, query_text=query.query_text, query_hash=query.query_hash)
query.latest_query_data = query_result
rv = self.make_request('get', '/api/queries/{}/results.json?api_key={}'.format(query.id, query.api_key), user=False)
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
def test_query_api_key_and_different_query_result(self):
ds = self.factory.create_data_source(group=self.factory.org.default_group, view_only=False)
query = self.factory.create_query(query_text="SELECT 8")
query_result2 = self.factory.create_query_result(data_source=ds, query_hash='something-different')
rv = self.make_request('get', '/api/queries/{}/results/{}.json?api_key={}'.format(query.id, query_result2.id, query.api_key), user=False)
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 404)
def test_signed_in_user_and_different_query_result(self):
ds2 = self.factory.create_data_source(group=self.factory.org.admin_group, view_only=False)
query = self.factory.create_query(query_text="SELECT 8")
query_result2 = self.factory.create_query_result(data_source=ds2, query_hash='something-different')
rv = self.make_request('get', '/api/queries/{}/results/{}.json'.format(query.id, query_result2.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
Unify query based dropdown population (#3337) * stop testing `collect_query_parameters`, it's an implementation detail * add tests for `missing_query_params` * rename SQLQuery -> ParameterizedSqlQuery * rename sql_query.py to parameterized_query.py * split to parameterized queries and parameterized SQL queries, where parameterized queries only do templating and parameterized SQL queries add tree validation on top of it * move missing parameter detection to ParameterizedQuery * get rid of some old code * fix tests * set syntax to `custom` * revert the max-age-related refactoring * 👋 tree validations 😢 * BaseQueryRunner is no longer a factory for ParameterizedQuery, for now * add an endpoint for running a query by its id and (optional) parameters without having to provide the query text * adds parameter schema to ParameterizedQuery * adds parameter schema validation (currently for strings) * validate number parameters * validate date parameters * validate parameters on POST /api/queries/<id>/results * validate enum parameters * validate date range parameters * validate query-based dropdowns by preprocessing them at the handler level and converting them to a populated enum * change _is_date_range to be a tad more succinct * a single assignment with a `map` is sufficiently explanatory * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/handlers/query_results.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * build error message inside the error * support all types of numbers as number parameters * check for permissions when populating query-based dropdowns * check for access to query before running it * check for empty rows when populating query-based enums * don't bother loading query results if user doesn't have access * 💥 on unexpected parameter types * parameter schema default is a list, not a dictionary * fix a totally unrelated typo * remove redundant null guards * introduce /dropdown.json endpoint with dummy data * wire frontend to /dropdown.json * always return name/value combos from /dropdown.json * load actual data into /dropdown.json * pluck correct values for `name` and `value` * reuse dropdwon plucking logic in QueryResultResource * simplify _get_dropdown_values * when doing parameter validation, we only care about the value and not the display name * rename dropdown to dropdownOptions * move dropdown_values to utils/parameterized_query.py * stop converting queries to enums and encapsulate the work inside ParameterizedQuery (almost - /dropdown.json would still access the dropdown_values method) * re-order arguments by importance * test query parameter validation * tests for dropdown_values logic * remove `.json` suffix to the dropdown endpoint * allow `BaseResource` to handle JSON stuff * move _pluck_name_and_value outside its containing method * case-insensitive lookup when plucking name and value * separate concerns and simplify test isolation for `dropdown_values` * pick the default column according to the order specified in the query result columns attribute * use `current_org` instead of passing `org` * test that user has access to the query when calling the /dropdown endpoint
2019-02-10 11:10:39 +00:00
class TestQueryResultDropdownResource(BaseTestCase):
def test_checks_for_access_to_the_query(self):
ds2 = self.factory.create_data_source(group=self.factory.org.admin_group, view_only=False)
query = self.factory.create_query(data_source=ds2)
rv = self.make_request('get', '/api/queries/{}/dropdown'.format(query.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
Unify query based dropdown population (#3337) * stop testing `collect_query_parameters`, it's an implementation detail * add tests for `missing_query_params` * rename SQLQuery -> ParameterizedSqlQuery * rename sql_query.py to parameterized_query.py * split to parameterized queries and parameterized SQL queries, where parameterized queries only do templating and parameterized SQL queries add tree validation on top of it * move missing parameter detection to ParameterizedQuery * get rid of some old code * fix tests * set syntax to `custom` * revert the max-age-related refactoring * 👋 tree validations 😢 * BaseQueryRunner is no longer a factory for ParameterizedQuery, for now * add an endpoint for running a query by its id and (optional) parameters without having to provide the query text * adds parameter schema to ParameterizedQuery * adds parameter schema validation (currently for strings) * validate number parameters * validate date parameters * validate parameters on POST /api/queries/<id>/results * validate enum parameters * validate date range parameters * validate query-based dropdowns by preprocessing them at the handler level and converting them to a populated enum * change _is_date_range to be a tad more succinct * a single assignment with a `map` is sufficiently explanatory * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/handlers/query_results.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * build error message inside the error * support all types of numbers as number parameters * check for permissions when populating query-based dropdowns * check for access to query before running it * check for empty rows when populating query-based enums * don't bother loading query results if user doesn't have access * 💥 on unexpected parameter types * parameter schema default is a list, not a dictionary * fix a totally unrelated typo * remove redundant null guards * introduce /dropdown.json endpoint with dummy data * wire frontend to /dropdown.json * always return name/value combos from /dropdown.json * load actual data into /dropdown.json * pluck correct values for `name` and `value` * reuse dropdwon plucking logic in QueryResultResource * simplify _get_dropdown_values * when doing parameter validation, we only care about the value and not the display name * rename dropdown to dropdownOptions * move dropdown_values to utils/parameterized_query.py * stop converting queries to enums and encapsulate the work inside ParameterizedQuery (almost - /dropdown.json would still access the dropdown_values method) * re-order arguments by importance * test query parameter validation * tests for dropdown_values logic * remove `.json` suffix to the dropdown endpoint * allow `BaseResource` to handle JSON stuff * move _pluck_name_and_value outside its containing method * case-insensitive lookup when plucking name and value * separate concerns and simplify test isolation for `dropdown_values` * pick the default column according to the order specified in the query result columns attribute * use `current_org` instead of passing `org` * test that user has access to the query when calling the /dropdown endpoint
2019-02-10 11:10:39 +00:00
Sharing embeds with safe parameters (#3495) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * support querystring params * extract coercing of numbers to function, along with a friendlier implementation * wire embeds to textless endpoint * allow users with view_only permissions to execute queries on the textless endpoint, as it only allows safe queries to run * enqueue jobs for ApiUsers * add parameters component for embeds * include existing parameters in embed code * fetch correct values for json requests * remove previous embed parameter code * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up) * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message on setting deprecation * rephrase deprecation message * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * change has_access and require_access signatures to work with the objects that require access, instead of their groups * split has_access between normal users and ApiKey users * remove residues from bad rebase * allow access to safe queries via api keys * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * fix bad rebase * send embed parameters through POST data * no need to log `is_api_key` * move query fetching by api_key to within the Query model * fetch user by adding a get_by_id function on the User model * pass parameters as POST data (fixes test failure introduced by switching from query string parameters to POST data) * test the right thing - queries with safe parameters should be embeddable * introduce cy.clickThrough * add another Cypress test to make sure unsafe queries cannot be embedded * serialize Parameters into query string * set is_api_key as the last parameter to (hopefully) avoid backward-dependency problems * Update redash/models/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * attempt to fix empty percy snapshots * snap percies after DOM is fully loaded
2019-04-02 08:45:38 +00:00
class TestQueryDropdownsResource(BaseTestCase):
def test_prevents_access_if_unassociated_and_doesnt_have_access(self):
Sharing embeds with safe parameters (#3495) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * support querystring params * extract coercing of numbers to function, along with a friendlier implementation * wire embeds to textless endpoint * allow users with view_only permissions to execute queries on the textless endpoint, as it only allows safe queries to run * enqueue jobs for ApiUsers * add parameters component for embeds * include existing parameters in embed code * fetch correct values for json requests * remove previous embed parameter code * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up) * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message on setting deprecation * rephrase deprecation message * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * change has_access and require_access signatures to work with the objects that require access, instead of their groups * split has_access between normal users and ApiKey users * remove residues from bad rebase * allow access to safe queries via api keys * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * fix bad rebase * send embed parameters through POST data * no need to log `is_api_key` * move query fetching by api_key to within the Query model * fetch user by adding a get_by_id function on the User model * pass parameters as POST data (fixes test failure introduced by switching from query string parameters to POST data) * test the right thing - queries with safe parameters should be embeddable * introduce cy.clickThrough * add another Cypress test to make sure unsafe queries cannot be embedded * serialize Parameters into query string * set is_api_key as the last parameter to (hopefully) avoid backward-dependency problems * Update redash/models/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * attempt to fix empty percy snapshots * snap percies after DOM is fully loaded
2019-04-02 08:45:38 +00:00
query = self.factory.create_query()
ds2 = self.factory.create_data_source(group=self.factory.org.admin_group, view_only=False)
unrelated_dropdown_query = self.factory.create_query(data_source=ds2)
# unrelated_dropdown_query has not been associated with query
# user does not have direct access to unrelated_dropdown_query
Sharing embeds with safe parameters (#3495) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * support querystring params * extract coercing of numbers to function, along with a friendlier implementation * wire embeds to textless endpoint * allow users with view_only permissions to execute queries on the textless endpoint, as it only allows safe queries to run * enqueue jobs for ApiUsers * add parameters component for embeds * include existing parameters in embed code * fetch correct values for json requests * remove previous embed parameter code * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up) * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message on setting deprecation * rephrase deprecation message * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * change has_access and require_access signatures to work with the objects that require access, instead of their groups * split has_access between normal users and ApiKey users * remove residues from bad rebase * allow access to safe queries via api keys * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * fix bad rebase * send embed parameters through POST data * no need to log `is_api_key` * move query fetching by api_key to within the Query model * fetch user by adding a get_by_id function on the User model * pass parameters as POST data (fixes test failure introduced by switching from query string parameters to POST data) * test the right thing - queries with safe parameters should be embeddable * introduce cy.clickThrough * add another Cypress test to make sure unsafe queries cannot be embedded * serialize Parameters into query string * set is_api_key as the last parameter to (hopefully) avoid backward-dependency problems * Update redash/models/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * attempt to fix empty percy snapshots * snap percies after DOM is fully loaded
2019-04-02 08:45:38 +00:00
rv = self.make_request('get', '/api/queries/{}/dropdowns/{}'.format(query.id, unrelated_dropdown_query.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
Sharing embeds with safe parameters (#3495) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * support querystring params * extract coercing of numbers to function, along with a friendlier implementation * wire embeds to textless endpoint * allow users with view_only permissions to execute queries on the textless endpoint, as it only allows safe queries to run * enqueue jobs for ApiUsers * add parameters component for embeds * include existing parameters in embed code * fetch correct values for json requests * remove previous embed parameter code * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up) * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message on setting deprecation * rephrase deprecation message * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * change has_access and require_access signatures to work with the objects that require access, instead of their groups * split has_access between normal users and ApiKey users * remove residues from bad rebase * allow access to safe queries via api keys * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * fix bad rebase * send embed parameters through POST data * no need to log `is_api_key` * move query fetching by api_key to within the Query model * fetch user by adding a get_by_id function on the User model * pass parameters as POST data (fixes test failure introduced by switching from query string parameters to POST data) * test the right thing - queries with safe parameters should be embeddable * introduce cy.clickThrough * add another Cypress test to make sure unsafe queries cannot be embedded * serialize Parameters into query string * set is_api_key as the last parameter to (hopefully) avoid backward-dependency problems * Update redash/models/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * attempt to fix empty percy snapshots * snap percies after DOM is fully loaded
2019-04-02 08:45:38 +00:00
def test_allows_access_if_unassociated_but_user_has_access(self):
query = self.factory.create_query()
query_result = self.factory.create_query_result()
data = {
'rows': [],
'columns': [{'name': 'whatever'}]
}
query_result = self.factory.create_query_result(data=json_dumps(data))
unrelated_dropdown_query = self.factory.create_query(latest_query_data=query_result)
# unrelated_dropdown_query has not been associated with query
# user has direct access to unrelated_dropdown_query
rv = self.make_request('get', '/api/queries/{}/dropdowns/{}'.format(query.id, unrelated_dropdown_query.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
def test_allows_access_if_associated_and_has_access_to_parent(self):
query_result = self.factory.create_query_result()
data = {
'rows': [],
'columns': [{'name': 'whatever'}]
}
query_result = self.factory.create_query_result(data=json_dumps(data))
dropdown_query = self.factory.create_query(latest_query_data=query_result)
options = {
'parameters': [{
'type': 'query',
'queryId': dropdown_query.id
}]
}
query = self.factory.create_query(options=options)
# dropdown_query has been associated with query
# user has access to query
rv = self.make_request('get', '/api/queries/{}/dropdowns/{}'.format(query.id, dropdown_query.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
def test_prevents_access_if_associated_and_doesnt_have_access_to_parent(self):
ds2 = self.factory.create_data_source(group=self.factory.org.admin_group, view_only=False)
dropdown_query = self.factory.create_query(data_source=ds2)
options = {
'parameters': [{
'type': 'query',
'queryId': dropdown_query.id
}]
}
query = self.factory.create_query(data_source=ds2, options=options)
# dropdown_query has been associated with query
# user doesnt have access to either query
rv = self.make_request('get', '/api/queries/{}/dropdowns/{}'.format(query.id, dropdown_query.id))
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 403)
Unify query based dropdown population (#3337) * stop testing `collect_query_parameters`, it's an implementation detail * add tests for `missing_query_params` * rename SQLQuery -> ParameterizedSqlQuery * rename sql_query.py to parameterized_query.py * split to parameterized queries and parameterized SQL queries, where parameterized queries only do templating and parameterized SQL queries add tree validation on top of it * move missing parameter detection to ParameterizedQuery * get rid of some old code * fix tests * set syntax to `custom` * revert the max-age-related refactoring * 👋 tree validations 😢 * BaseQueryRunner is no longer a factory for ParameterizedQuery, for now * add an endpoint for running a query by its id and (optional) parameters without having to provide the query text * adds parameter schema to ParameterizedQuery * adds parameter schema validation (currently for strings) * validate number parameters * validate date parameters * validate parameters on POST /api/queries/<id>/results * validate enum parameters * validate date range parameters * validate query-based dropdowns by preprocessing them at the handler level and converting them to a populated enum * change _is_date_range to be a tad more succinct * a single assignment with a `map` is sufficiently explanatory * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/handlers/query_results.py Co-Authored-By: rauchy <omer@rauchy.net> * Update redash/utils/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * build error message inside the error * support all types of numbers as number parameters * check for permissions when populating query-based dropdowns * check for access to query before running it * check for empty rows when populating query-based enums * don't bother loading query results if user doesn't have access * 💥 on unexpected parameter types * parameter schema default is a list, not a dictionary * fix a totally unrelated typo * remove redundant null guards * introduce /dropdown.json endpoint with dummy data * wire frontend to /dropdown.json * always return name/value combos from /dropdown.json * load actual data into /dropdown.json * pluck correct values for `name` and `value` * reuse dropdwon plucking logic in QueryResultResource * simplify _get_dropdown_values * when doing parameter validation, we only care about the value and not the display name * rename dropdown to dropdownOptions * move dropdown_values to utils/parameterized_query.py * stop converting queries to enums and encapsulate the work inside ParameterizedQuery (almost - /dropdown.json would still access the dropdown_values method) * re-order arguments by importance * test query parameter validation * tests for dropdown_values logic * remove `.json` suffix to the dropdown endpoint * allow `BaseResource` to handle JSON stuff * move _pluck_name_and_value outside its containing method * case-insensitive lookup when plucking name and value * separate concerns and simplify test isolation for `dropdown_values` * pick the default column according to the order specified in the query result columns attribute * use `current_org` instead of passing `org` * test that user has access to the query when calling the /dropdown endpoint
2019-02-10 11:10:39 +00:00
Sharing embeds with safe parameters (#3495) * change has_access and require_access signatures to work with the objects that require access, instead of their groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * use the textless endpoint (/api/queries/:id/results) for pristine queriest * Revert "use the textless endpoint (/api/queries/:id/results) for pristine" This reverts commit cd2cee77385ecf79efd2f1aa21fab0dd43943264. * go to textless /api/queries/:id/results by default * change `run_query`'s signature to accept a ParameterizedQuery instead of constructing it inside * raise HTTP 400 when receiving invalid parameter values. Fixes #3394 * support querystring params * extract coercing of numbers to function, along with a friendlier implementation * wire embeds to textless endpoint * allow users with view_only permissions to execute queries on the textless endpoint, as it only allows safe queries to run * enqueue jobs for ApiUsers * add parameters component for embeds * include existing parameters in embed code * fetch correct values for json requests * remove previous embed parameter code * rename `id` to `user_id` * support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query * bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up) * show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move other message (email not verified) to use the same mechanism * add link to forum message on setting deprecation * rephrase deprecation message * add link to forum message regarding embed deprecation * change API to /api/queries/:id/dropdowns/:dropdown_id * split to 2 different dropdown endpoints and implement the second * add test cases for /api/queries/:id/dropdowns/:id * use new /dropdowns endpoint in frontend * first e2e test for sharing embeds * Pleasing the CodeClimate overlords * All glory to CodeClimate * change has_access and require_access signatures to work with the objects that require access, instead of their groups * split has_access between normal users and ApiKey users * remove residues from bad rebase * allow access to safe queries via api keys * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * change has_access and require_access signatures to work with the objects that require access, instead of their groups * rename `object` to `obj` * support both objects and group dicts in `has_access` and `require_access` * simplify permission tests once `has_access` accepts groups * fix bad rebase * send embed parameters through POST data * no need to log `is_api_key` * move query fetching by api_key to within the Query model * fetch user by adding a get_by_id function on the User model * pass parameters as POST data (fixes test failure introduced by switching from query string parameters to POST data) * test the right thing - queries with safe parameters should be embeddable * introduce cy.clickThrough * add another Cypress test to make sure unsafe queries cannot be embedded * serialize Parameters into query string * set is_api_key as the last parameter to (hopefully) avoid backward-dependency problems * Update redash/models/parameterized_query.py Co-Authored-By: rauchy <omer@rauchy.net> * attempt to fix empty percy snapshots * snap percies after DOM is fully loaded
2019-04-02 08:45:38 +00:00
class TestQueryResultExcelResponse(BaseTestCase):
def test_renders_excel_file(self):
query = self.factory.create_query()
query_result = self.factory.create_query_result()
rv = self.make_request('get', '/api/queries/{}/results/{}.xlsx'.format(query.id, query_result.id), is_json=False)
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)
def test_renders_excel_file_when_rows_have_missing_columns(self):
query = self.factory.create_query()
data = {
'rows': [
{'test': 1},
{'test': 2, 'test2': 3},
],
'columns': [
{'name': 'test'},
{'name': 'test2'},
],
}
query_result = self.factory.create_query_result(data=json_dumps(data))
rv = self.make_request('get', '/api/queries/{}/results/{}.xlsx'.format(query.id, query_result.id), is_json=False)
Migrate the application to Python 3 (#4251) * Make core app compatible with Python 3 No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary. * Use Python 3.7 as base docker image Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here. * Upgrade some requirements to newest versions Some of the older versions were not compatible with Python 3. * Migrate tests to Python 3 * Build frontend on Python 3 * Make the HMAC sign function compatible with Python 3 In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures. * Use assertCountEqual instead of assertItemsEqual The latter is not available in Python 3. See https://bugs.python.org/issue17866 * Remove redundant encoding header for Python 3 modules * Remove redundant string encoding in CLI * Rename list() functions in CLI These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes. * Replace usage of Exception.message in CLI `message` is not available anymore, instead use the string representation of the exception. * Adapt test handlers to Python 3 * Fix test that relied on dict ordering * Make sure test results are always uploaded (#4215) * Support encoding memoryview to JSON psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156 * Fix test relying on object address ordering * Decode bytes returned from Redis * Stop using e.message for most exceptions Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries. * Fix writing XLSX files in Python 3 The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue. * Fix test by comparing strings to strings * Fix another exception message unavailable in Python 3 * Fix export to CSV in Python 3 The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`. * (Python 3) Use Redis' decode_responses=True option (#4232) * Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL * Fix displaying error while connecting to SQLite The exception message is always a string in Python 3, so no need to try to decode things. * Fix another missing exception message * Handle JSON encoding for datasources returning bytes SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data. * Fix Python 3 compatibility with RQ * Revert some changes 2to3 tends to do (#4261) - Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning. * Upgrade dependencies * Remove useless `iter` added by 2to3 * Fix get_next_path tests (#4280) * Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug * Switched pytz and certifi to unbinded versions. * Switch to new library for getting country from IP `python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package . * Python 3 RQ modifications (#4281) * show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions * Remove legacy Python flake8 tests
2019-10-24 09:42:13 +00:00
self.assertEqual(rv.status_code, 200)