The Archive module allows you to create compressed data files (zstd) from Elasticsearch indexes. The archive checks the age of each document in the index and if it is older than defined in the job, it is copied to the archive file.


Enabling module

To configure module edit kibana.yml configuration file end set path to the archive directory - location where the archive files will be stored:

vim /etc/kibana/kibana.yml

remove the comment from the following line and set the correct path to the archive directory:

archive.archivefolderpath: '/var/lib/elastic_archive_test'

Archive Task

Create Archive task

  1. From the main navigation go to the “Archvie” module.


  2. On the “Archive” tab select “Create Task” and define the following parameters:

    • Index pattern- for the indexes that will be archive, for example syslog* ;
    • Older than (days) - number of days after which documents will be archived;
    • Schedule task (crontab format) - the work schedule of the ordered task.


Task List

In the Task List you can follow the current status of ordered tasks. You can modify task scheduler or delete ordered task.


If the archiving task finds an existing archive file that matches the data being archived, it will check the number of documents in the archive and the number of documents in the index. If there is a difference in the number of documents then new documents will be added to the archive file.

Archive Upload

The Archive Upload module move data from archive to Elasticsearch index and make it online.

Create Upload task

  1. From the main navigation go to the Archive module.
  2. On the Upload tab select Create Task and define the following parameters:
    • Destination index - If destination index does not exist it will be created. If exists data will append.
  • File name - list of archive file that will be recover to Elasticsearch index.


Task List

The process will index data back into Elasticsearch. Depend on archive size the process can take long time. On the Task List you can follow the status of the recovery process. Also you can view result and delete tasks.


Command Line tools

Archive files can be handled by the following commands zstd, zstdcat, zstdgrep, zstdless, zstdmt.


The command for decompress *.zstd the Archive files, for example:

zstd -d winlogbeat-2020.10_2020-10-23.json.zstd -o


The command for concatenate *.zstd Archive files and print content on the standard output, for example:

zstdcat winlogbeat-2020.10_2020-10-23.json.zstd


The command for print lines matching a pattern from *.zstd Archive files, for example:

zstdgrep "optima" winlogbeat-2020.10_2020-10-23.json.zstd

Above example is searching documents contain the “optima” phrase in winlogbeat-2020.10_2020-10-23.json.zstd archive file.


The command for viewing Archive * .zstd files, for example:

zstdless winlogbeat-2020.10_2020-10-23.json.zstd


The command for compress and decompress Archive *.zdtd file useing multiple CPU core (default is 1), for example:

zstdmt -d winlogbeat-2020.10_2020-10-23.json.zstd -o winlogbeat-2020.10_2020-10-23.json