valitydev/redash
14
0
Fork 0
mirror of https://github.com/valitydev/redash.git synced 2024-11-07 09:28:51 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Use a paginator when iterating over glue tables

Browse Source 
Without this you only get the first 100 tables in the glue database
This commit is contained in:
Blake Imsland 2018-02-08 13:02:53 -08:00 committed by Jannis Leidel
parent 570187fc1f
commit dceb58cd79
No known key found for this signature in database
GPG Key ID: C795956FB489DCA9
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
redash/query_runner/athena.py
View File

@ -125,9 +125,11 @@ class Athena(BaseQueryRunner):
region_name=self.configuration['region'] region_name=self.configuration['region']
) )
schema = {} schema = {}
paginator = client.get_paginator('get_tables')
for database in client.get_databases()['DatabaseList']: for database in client.get_databases()['DatabaseList']:
for table in client.get_tables(DatabaseName=database['Name'])['TableList']: iterator = paginator.paginate(DatabaseName=database['Name'])
for table in iterator.search('TableList[]'):
table_name = '%s.%s' % (database['Name'], table['Name']) table_name = '%s.%s' % (database['Name'], table['Name'])
if table_name not in schema: if table_name not in schema:
column = [columns['Name'] for columns in table['StorageDescriptor']['Columns']] column = [columns['Name'] for columns in table['StorageDescriptor']['Columns']]