======= Modules ======= ******** TankCore ******** Core class. Represents basic steps of test execution. Simplifies plugin configuration, configs reading, artifacts storing. Represents parent class for modules/plugins. INI file section: **[tank]** Architecture ============ .. image:: ./pic/tank-architecture.png Test lifecycle ============== .. image:: ./pic/tank-lifecycle.png Options ======= Basic options: :lock_dir: Directory for lockfile. Default: ``/var/lock/``. :plugin_: Path to plugin. Empty path interpreted as disable of plugin. :artifacts_base_dir: Base directory for artifacts storing. Temporary artifacts files are stored here. Default: current directory. :artifacts_dir: Directory where to keep artifacts after test. Default: directory in ``artifacts_base_dir`` named in Date/Time format. :flush_config_to: Dump configuration options after each tank step (`yandex.tank steps. sorry, russian only `_) to that file :taskset_path: Path to taskset command. Default: taskset. :affinity: Set a yandex-tank's (python process and load generator process) CPU affinity. Default: empty. Example: ``0-3`` enabling first 4 cores, '0,1,2,16,17,18' enabling 6 cores. consoleworker ============= Consoleworker is a cmd-line interface for Yandex.Tank. Worker class that runs and configures TankCore accepting cmdline parameters. Human-friendly unix-way interface for yandex-tank. Command-line options described above. apiworker ========= apiworker is a python interface for Yandex.Tank. Worker class for python. Runs and configures TankCore accepting ``dict()``. Python-frinedly interface for yandex-tank. Example: .. code-block:: python from yandextank.api.apiworker import ApiWorker import logging import traceback import sys logger = logging.getLogger('') logger.setLevel(logging.DEBUG) #not mandatory options below: options = dict() options['config'] = '/path/to/config/load.ini' options['manual_start'] = "1" options['user_options'] = [ 'phantom.ammofile=/path/to/ammofile', 'phantom.rps_schedule=const(1,2m)', ] log_filename = '/path/to/log/tank.log' #====================================== apiworker = ApiWorker() apiworker.init_logging(log_filename) try: apiworker.configure(options) apiworker.perform_test() except Exception, ex: logger.error('Error trying to perform a test: %s', ex) exit codes ========== .. code-block:: json { "0": "completed", "1": "interrupted_generic_interrupt", "2": "interrupted", "3": "interrupted_active_task_not_found ", "4": "interrupted_no_ammo_file", "5": "interrupted_address_not_specified", "6": "interrupted_cpu_or_disk_overload", "7": "interrupted_unknown_config_parameter", "8": "interrupted_stop_via_web", "9": "interrupted", "11": "interrupted_job_number_error", "12": "interrupted_phantom_error", "13": "interrupted_job_metainfo_error", "14": "interrupted_target_monitoring_error", "15": "interrupted_target_info_error", "21": "autostop_time", "22": "autostop_http", "23": "autostop_net", "24": "autostop_instances", "25": "autostop_total_time", "26": "autostop_total_http", "27": "autostop_total_net", "28": "autostop_negative_http", "29": "autostop_negative_net", "30": "autostop_http_trend", "31": "autostop_metric_higher", "32": "autostop_metric_lower" } *************** Load Generators *************** Phantom ======= Load generator module that uses phantom utility. INI file section: **[phantom]** How it works ------------ .. image:: ./pic/tank-stepper.png Options ------- Basic options ^^^^^^^^^^^^^ :ammofile: Ammo file path (ammo file is a file containing requests that are to be sent to a server. Could be gzipped). :rps_schedule: Load schedule in terms of RPS. :instances: Max number of instances (concurrent requests). :instances_schedule: Load schedule in terms of number of instances. :loop: Number of times requests from ammo file are repeated in loop. :ammo_limit: Limit request number. :autocases: Enable marking requests automatically. Available options: 1 -- enable, 0 -- disable). :chosen_cases: Use only selected cases. There are 3 ways to constrain requests number: by schedule with ``rps_schedule``, by requests number with ``ammo_limit`` or by loop number with ``loop`` option. Tank stops if any constrain is reached. If stop reason is reached ``ammo_limit`` or ``loop`` it will be mentioned in log file. In test without ``rps_schedule`` file with requests is used one time by default. Additional options ^^^^^^^^^^^^^^^^^^ :writelog: Enable verbose request/response logging. Default: 0. Available options: 0 - disable, all - all messages, proto_warning - 4хх+5хх+network errors, proto_error - 5хх+network errors. :ssl: Enable SSL. Default: 0. Available options: 1 - enable, 0 - disable. :timeout: Response timeout. Default: ``11s``. .. note:: Default multiplier is ``seconds``. If you specify ``10``, timeout will be 10 seconds. Currently we support here multipliers: 'd' for days, 'h' for hours, 'm' for minutes, 's' for seconds Examples: ``0.1s`` is 100 milliseconds. ``1m`` for 1 minute. :address: Address of target. Default: ``127.0.0.1``. Format: ``[host]:port``, ``[ipv4]:port``, ``[ipv6]:port``. Tank checks each test if port is available. :port (deprecated, use ``address``): Port of target. Default: ``80``. :gatling_ip: Use multiple source addresses. List, divided by spaces. :tank_type: Available options: ``http`` and ``none`` (raw TCP). Default: ``http``. :eta_file: Path to ETA file. :connection_test: Test TCP socket connection before starting the test. Default: 1. Available options: 1 - enable, 0 - disable. URI-style options ^^^^^^^^^^^^^^^^^ :uris: URI list, multiline option. :headers: HTTP headers list in the following form: ``[Header: value]``, multiline option. :header\_http: HTTP version. Default: ``1.0`` Available options: ``1.0`` and ``1.1``. ``2.0`` is NOT supported by this load generator. stpd-file cache options ^^^^^^^^^^^^^^^^^^^^^^^ :use_caching: Enable cache. Default: ``1``. :cache_dir: Cache files directory. Default: base artifacts directory. :force_stepping: Force stpd file generation. Default: ``0``. Advanced options ^^^^^^^^^^^^^^^^ :phantom_path: Phantom utility path. Default: ``phantom``. :phantom_modules_path: Phantom modules path. Default: ``/usr/lib/phantom``. :config: Use given (in this option) config file for phantom instead of generated. :phout_file: Import this phout instead of launching phantom (import phantom results). :stpd_file: Use this stpd-file instead of generated. :threads: Phantom thread count. Default: ``/2 + 1``. :buffered_seconds: Amount of seconds to which delay aggregator, to be sure that everything were read from phout. :additional_libs: List separated by whitespaces, will be added to phantom config file in section ``module_setup`` :method_prefix: Object's type, that has a functionality to create test requests. Default: ``method_stream``. :source_log_prefix: Prefix, added to class name that reads source data. Default: empty. :method_options: Additional options for method objects. It is used for Elliptics etc. Default: empty. :affinity: Set a phantom's CPU affinity. Example: ``0-3`` enabling first 4 cores, '0,1,2,16,17,18' enabling 6 cores. Default: empty. TLS/SSL additional options ^^^^^^^^^^^^^^^^^^^^^^^^^^ .. note:: ``ssl=1`` is required :client_cipher_suites: Cipher list, consists of one or more cipher strings separated by colons (see man ciphers). Example: client_cipher_suites = RSA:!COMPLEMENTOFALL Default: empty. :client_certificate: Path to client certificate which is used in client's "Certificate message" in Client-authenticated TLS handshake. Default: empty. :client_key: Path to client's certificate's private key, used for client's "CertificateVerify message" generation in Client-authenticated TLS handshake. Default: empty. Phantom http-module tuning options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :phantom_http_line: First line length. Default: ``1K``. :phantom_http_field_num: Headers amount. Default: ``128``. :phantom_http_field: Header size. Default: ``8K``. :phantom_http_entity: Answer ``size``. Default: ``8M``. .. note:: Please, keep in mind, especially if your service has large answers, that phantom doesn't read more than defined in ``phantom_http_entity``. Artifacts --------- :phantom_*.conf: Generated configuration files. :phout_*.log: Raw results file. :phantom_stat_*.log: Phantom statistics, aggregated by seconds. :answ_*.log: Detailed request/response log. :phantom_*.log: Internal phantom log. Multi-tests ----------- To make several simultaneous tests with phantom, add proper amount of sections with names ``phantom-_N_``. All subtests are executed in parallel. Multi-test ends as soon as one subtest stops. Example: :: [phantom] phantom_path=phantom ammofile=data/dummy.ammo instances=10 instances_schedule=line(1,10,1m) loop=1 use_caching=1 [phantom-1] uris=/ /test /test2 headers=[Host: www.ya.ru] [Connection: close] rps_schedule=const(1,30) line(1,1000,2m) const(1000,5m) address=fe80::200:f8ff:fe21:67cf port=8080 ssl=1 timeout=15 instances=3 gatling_ip=127.0.0.1 127.0.0.2 phantom_http_line=123M [phantom-2] uris=/3 rps_schedule=const(1,30) line(1,50,2m) const(50,5m) Options that apply only for main section: buffered_seconds, writelog, phantom_modules_path, phout_file, config, eta_file, phantom_path JMeter ====== JMeter module uses JMeter as a load generator. To enable it, disable phantom first (unless you really want to keep it active alongside at your own risk), enable JMeter plugin and then specify the parameters for JMeter: :: [tank] ; Disable phantom: plugin_phantom= ; Enable JMeter instead: plugin_jmeter=yandextank.plugins.JMeter INI file section: **[jmeter]** Options ------- :jmx: Testplan for execution. :args: Additional commandline arguments for JMeter. :jmeter_path: Path to JMeter, allows to use alternative JMeter installation. Default: ``jmeter`` :buffered_seconds: Amount of seconds to which delay aggregator, to be sure that everything were read from jmeter's results file. :jmeter_ver: Which jmeter version tank should expect. Currently it affects the way connection time is logged, but may be used for other version-specific settings. Default: ``3.0`` :ext_log: Available options: ``none``, ``errors``, ``all``. Add one more simple data writer which logs all possible fields in jmeter xml format, this log is saved in test dir as ``jmeter_ext_XXXX.jtl``. Default: ``none`` :all other options in the section: They will be passed as User Defined Variables to JMeter. Timing calculation issues ----------------------- Since version 2.13 jmeter could measure connection time, latency and full request time (aka in phantom), but do it in it's own uniq way: latency include connection time but not recieve time. For the sake of consistency we recalculate as and calculate as >, but it does not guranteed to work perfectly in all cases (i.e. some samplers may not support latency and connect_time and you may get something strange in case of timeouts). For jmeter 2.12 and older connection time logging not avaliable, set ``jmeter_ver`` properly or you'll get an error for unknown field in Simlpe Data Writer listner added by tank. Artifacts --------- :: Original testplan. :: Modified test plan with results output section. :: JMeter's results. :: JMeter's log. BFG === (`What is BFG `_) BFG is a generic gun that is able to use different kinds of cannons to shoot. To enable it, disable phantom first (unless you really want to keep it active alongside at your own risk), enable BFG plugin and then specify the parameters for BFG and for the gun of your choice. There are three predefined guns: Log Gun, Http Gun and SQL gun. First two are mostly for demo, if you want to implement your own gun class, use them as an example. But the main purpose of BFG is to support user-defined scenarios in python. Here is how you do it using 'ultimate' gun. 1. Define your scenario as a python class (in a single-file module, or a package): .. code-block:: python import logging log = logging.getLogger(__name__) class LoadTest(object): def __init__(self, gun): # you'll be able to call gun's methods using this field: self.gun = gun # for example, you can get something from the 'ultimate' section of a config file: my_var = self.gun.get_option("my_var", "hello") def case1(self, missile): # we use gun's measuring context to measure time. # The results will be aggregated automatically: with self.gun.measure("case1"): log.info("Shoot case 1: %s", missile) # there could be multiple steps in one scenario: with self.gun.measure("case1_step2") as sample: log.info("Shoot case 1, step 2: %s", missile) # and we can set the fields of measured object manually: sample["proto_code"] = 500 # the list of available fields is below def case2(self, missile): with self.gun.measure("case2"): log.info("Shoot case 2: %s", missile) def setup(self, param): ''' this will be executed in each worker before the test starts ''' log.info("Setting up LoadTest: %s", param) def teardown(self): ''' this will be executed in each worker after the end of the test ''' log.info("Tearing down LoadTest") #It's mandatory to explicitly stop worker process in teardown os._exit(0) return 0 2. Define your options in a config file: :: [tank] ; Disable phantom: plugin_phantom= ; Enable BFG instead: plugin_bfg=yandextank.plugins.Bfg [bfg] ; parallel processes count instances = 10 ; gun type gun_type = ultimate ; ammo file ammofile=req_json.log ; load schedule rps_schedule=line(1,100,10m) [ultimate_gun] ; path to your custom module module_path = ./my_own_service ; python module name module_name = mygun ; gun initialization parameter init_param = Hello 3. Create an ammo file: Ammo format: one line -- one request, each line begins with case name separated by tab symbol ('\t'). Case name defines the method of your test class that will be executed. The line itself will be passed to your method as 'missile' parameter. If there was no case name for an ammo, the 'default' case name will be used :: case1my-case1-ammo case2my-case2-ammo my-default-case-ammo .. note:: TIP: if each line is a JSON-encoded document, you can easily parse it inside your scenario code 4. Shoot em all! How it works ------------ .. image:: ./pic/tank-bfg.png BFG Worker Type ----------- By default, BFG will create lots of processes (number is defined by ``instances`` option). Every process will execute requests in a single thread. These processes will comsume a lot of memory. It's also possible to switch this behavior and use ``gevent`` to power up every worker process, allowing it to have multiple concurrent threads executing HTTP requests. With green worker, it's recommended to set ``instances`` to number of CPU cores, and adjust the number of real threads by ``green_threads_per_instance`` option. INI file section: **[bfg]** :worker_type: Set it to ``green`` to let every process have multiple concurrent green threads. :green_threads_per_instance: Number of green threads every worker process will execute. Only affects ``green`` worker type. BFG Options ----------- INI file section: **[bfg]** :gun_type: What kind of gun should BFG use. :ammo_type: What ammo parser should BFG use. Default: ``caseline``. :pip: Install python modules with ``pip install --user`` before the test. If you need multiple modules use multiline options, i.e.: :: pip=grequests msgpack :init_param: An initialization parameter that will be passed to your ``setup`` method. :other common stepper options: Ultimate Gun Options ------------------ gun_type = **ultimate** INI file section: **[ultimate_gun]** :module_path: Path to your module :module_name: Python module name :class_name: Class that contains load scenarios, default: LoadTest The fields of measuring context object and their default values: :send_ts: A timestamp when context was entered. :tag: A marker passed to the context. :interval_real: The time interval from enter to exit. If the user defines his own value, it will be preserved. Microseconds. :connect_time: Microseconds. Default: 0 :send_time: Microseconds. Default: 0 :latency: Microseconds. Default: 0 :receive_time: Microseconds. Default: 0 :interval_event: Microseconds. Default: 0 :size_out: Bytes out. Integer. Default: 0 :size_in: Bytes in. Integer. Default: 0 :net_code: Network code. Integer. Default: 0 :proto_code: Protocol code (http, for example). Integer. Default: 200 SQL Gun Options --------------- gun_type = **sql** INI file section: **[sql_gun]** :db: DB uri in format: ``dialect+driver://user:password@host/dbname[?key=value..]``, where dialect is a database name such as mysql, oracle, postgresql, etc., and driver the name of a DBAPI, such as psycopg2, pyodbc, cx_oracle, etc. `details `_ Pandora ======= `Pandora `_ is a load generator written in Go. For now it supports only SPDY/3 and HTTP(S). Plugins for other protocols (HTTP/2, Websocket, XMPP) are on the way. First of all you'll need to obtain a binary of pandora and place it somewhere on your machine. By default, Yandex.Tank will try to just run ``pandora`` (or you could specify a path to binary in ``pandora_cmd``). Disable phantom first (unless you really want to keep it active alongside at your own risk), enable Pandora plugin and then specify the parameters. :: [tank] ; Disable phantom: plugin_phantom= ; Enable Pandora instead: plugin_pandora=yandextank.plugins.Pandora ; Pandora config section: [pandora] ; Pandora executable path pandora_cmd=/usr/bin/pandora ; Enable/disable expvar monitoring expvar = 1 ; default ; Pandora config contents (json) config_content = { "pools": [ { "name": "dummy pool", "gun": {"type": "log"}, "ammo": { "type": "dummy/log", "AmmoLimit": 10000000 }, "result": { "type": "log/phout", "destination": "./phout.log" }, "shared-limits": false, "user-limiter": { "type": "unlimited" }, "startup-limiter": { "type": "periodic", "batch": 1, "max": 5, "period": "0.5s" } }]} ; OR config file (yaml or json) config_file = pandora_config.yml Schedules --------- The first schedule type is ``periodic`` schedule. It is defined as ``periodic(, , )``. Pandora will issue one batch of size ``batch_size``, once in ``period`` seconds, maximum of ``limit`` ticks. Those ticks may be used in different places, for example as a limiter for user startups or as a limiter for each user request rate. Example: :: startup_schedule = periodic(2, 0.1, 100) user_schedule = periodic(10, 15, 100) shared_schedule = 0 Start 2 users every 0.1 seconds, 100 batches, maximum of 2 * 100 = 200 users. Each user will issue requests in batches of 10 requests, every 15 seconds, maximum of 100 requests. All users will read from one ammo source. Second schedule type is ``linear``. It is defined like this: ``linear(, ,