Agents module¶

The Agents module is used for the central management of agents used in Energy Logserver such as Filebeat, Winlogbeat, Packetbeat, Metricbeat, Logstash and all other configuration files.

Component modules¶

The software consists of two modules:

Agent Module - installation just like any standard Kibana plugin.
Agent software - installed on host with agent (like beats);

Agent Module installation¶

All necessary components can be found in the installation folder ./install/Agents/masteragent.

Go to installation directory:

cd ./install/Agents/masteragent

Generating the certificates:
```
cd certificates/
```
- set DOMAIN and DOMAIN_IP in scripts from ./certificates directory:
```
#!/bin/bash
DOMAIN="localhost"
DOMAIN_IP="10.4.3.185"
```
- execute the scripts in the following order:
```
./1_rootca.sh
./2_clientcrt.sh
./3_createstore.sh
```
Install the required packages:
```
yum install net-tools
```

Add an exception to the firewall to listen on TCP 8080 and 8081:

firewall-cmd --permanent --zone public --add-port 8080/tcp
firewall-cmd --permanent --zone public --add-port 8081/tcp

Logstash pipeline configuration:

/bin/cp -rf ./logstash/agents_template.json /etc/logstash/templates.d/
mkdir /etc/logstash/conf.d/masteragent
/bin/cp -rf ./logstash/*.conf /etc/logstash/conf.d/masteragent/

Edit file /etc/logstash/pipelines.yml by uncomment this line(be awer to this lines looks likes other uncomment lines. It’s yml file.):

- pipeline.id: masteragent
  path.config: "/etc/logstash/conf.d/masteragent/*.conf

Logstash SSL configuration:

mkdir /etc/logstash/conf.d/masteragent/ssl
/bin/cp -rf ./certificates/domain.key /etc/logstash/conf.d/masteragent/ssl/
/bin/cp -rf ./certificates/domain.crt /etc/logstash/conf.d/masteragent/ssl/
/bin/cp -rf ./certificates/rootCA.crt /etc/logstash/conf.d/masteragent/ssl/
chown -R logstash:logstash /etc/logstash

Linux Agent installation¶

Copy necessary files to destination host:

/bin/cp -rf ./install/Agents/masteragent/agents/linux/masteragent /opt/masteragent
/bin/cp -rf ./install/Agents/masteragent/certificates/node_name.p12 /opt/masteragent
/bin/cp -rf ./install/Agents/masteragent/certificates/root.jks /opt/masteragent
/bin/cp -rf    ./install/Agents/masteragent/agents/linux/masteragent/masteragent.service /usr/lib/systemd/system/masteragent.service

Set correct IP address of Logstash and Kibana in /opt/masteragent/agent.conf and verify paths for Filebeat, Metricbeat, etc. are correct.
```
systemctl daemon-reload
systemctl enable masteragent
systemctl start masteragent
```
Restart logstash:
```
systemctl restart logstash
```
In the GUI, in the Agents tab, you can check the status of the newly connected host.

Windows Agent installation¶

Add an exception to the firewall to listen on TCP port 8081.
Add an exception to the firewall enabling connection on TCP LOGSTASH_IP:8080 port.
Copy content of the ./agents/windows from installation directory to “C:\Program Files\MasterAgnet”
Change IP address of the Kibana GUI server and Logstash server in “C:\Program Files\MasterAgnet\agent.conf” file.
In order to install the service, start the console as an administrator and execute the following commands:
```
cd "C:\Program Files\MasterAgent"
agents.exe install
agents.exe start
```
An alternative method of installing the service, run the PowerShell console as administrator and execute the following commands:
```
New-Service -name masteragent -displayName masteragent - binaryPathName "C:\Program Files\MasterAgent\agents.exe"
```
Check status of service via services.msc (if stoped, try start it agian).
In the GUI, in the Agents tab, you can check the status of the newly connected host.

Agent module compatibility¶

The Agents module works with Beats agents in the following versions:

Nr	Agent Name	Beats Version	Link to download
1	Filebeat	OSS 6.8.14	https://www.elastic.co/downloads/past-releases/filebeat-oss-6-8-13
2	Packetbeat	OSS 6.8.14	https://www.elastic.co/downloads/past-releases/packetbeat-oss-6-8-13
3	Winlogbeat	OSS 6.8.14	https://www.elastic.co/downloads/past-releases/winlogbeat-oss-6-8-13
4	Metricbeat	OSS 6.8.14	https://www.elastic.co/downloads/past-releases/metricbeat-oss-6-8-13
5	Heartbeat	OSS 6.8.14	https://www.elastic.co/downloads/past-releases/heartbeat-oss-6-8-13
6	Auditbeat	OSS 6.8.14	https://www.elastic.co/downloads/past-releases/auditbeat-oss-6-8-13
7	Logstash	OSS 6.8.14	https://www.elastic.co/downloads/past-releases/logstash-oss-6-8-13

Beats agents installation¶

Windows¶

Winlogbeat¶

Installation¶

Copy the Winlogbeat installer from the installation directory install/Agents/beats/windows/winlogbeat-oss-6.8.14-windows-x86_64.zip and unpack
Copy the installation files to the C:\Program Files\Winlogbeat directory

Configuration¶

Editing the file: C:\Program Files\Winlogbeat\winlogbeat.yml:

In section:

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
  - name: System

change to:

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
    ignore_older: 72h
  - name: System
    ignore_older: 72h

In section:

setup.template.settings:
  index.number_of_shards: 1

change to:

#setup.template.settings:
  #index.number_of_shards: 1

In section:
```
setup.kibana:
```
change to:
```
#setup.kibana:
```

In section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

change to:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

In section:

#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

change to:

output.logstash:
  # The Logstash hosts
  hosts: ["LOGSTASH_IP:5044"]

In section:

#tags: ["service-X", "web-tier"]

change to:

tags: ["winlogbeat"]

Run the PowerShell console as Administrator and execute the following commands:

cd 'C:\Program Files\Winlogbeat'
.\install-service-winlogbeat.ps1

Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Winlogbeat\install-service-winlogbeat.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R

Output:

Status   Name               DisplayName
------   ----               -----------
Stopped    Winlogbeat      Winlogbeat

Start Winlogbeat service:

sc start Winlogbeat

Test configuration:

cd 'C:\Program Files\Winlogbeat'
winlogbeat.exe test config
winlogbeat.exe test output

Filebeat¶

Installation¶

Copy the Filebeat installer from the installation directory install/Agents/beats/windows/filebeat-oss-6.8.14-windows-x86_64.zip and unpack
Copy the installation files to the C:\Program Files\Filebeat directory

Configuration¶

Editing the file: C:\Program Files\Filebeat\filebeat.yml:

In section:

- type: log

  # Change to true to enable this input configuration.
  enabled: false

change to:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

In section:

    paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

change to:

paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - "C:\Program Files\Microsoft SQL Server\*\MSSQL\Log\*"
    - "C:\inetpub\logs\*""

In section:

setup.template.settings:
  index.number_of_shards: 1

change to:

#setup.template.settings:
  #index.number_of_shards: 1

In section:
```
setup.kibana:
```
change to:
```
#setup.kibana:
```

In section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

change to:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

In section:

#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

change to:

output.logstash:
  # The Logstash hosts
  hosts: ["LOGSTASH_IP:5044"]

In section:

#tags: ["service-X", "web-tier"]

change to:

tags: ["filebeat"]

Run the PowerShell console as Administrator and execute the following commands:

cd 'C:\Program Files\Filebeat'
.\install-service-filebeat.ps1

Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Filebeat\install-service-filebeat.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R

Output:

Status   Name               DisplayName
------   ----               -----------
Stopped  Filebeat        Filebeat

Start Filebeat service:

sc start filebeat

You can enable, disable and list Filebeat modules using the following command:

cd 'C:\Program Files\Filebeat'
filebeat.exe modules list
filebeat.exe modules apache enable
filebeat.exe modules apache disable

Test configuration:

cd 'C:\Program Files\Filebeat'
filebeat.exe test config
filebeat.exe test output

Merticbeat¶

Installation¶

Copy the Merticbeat installer from the installation directory install/Agents/beats/windows/merticbeat-oss-6.8.14-windows-x86_64.zip and unpack
Copy the installation files to the C:\Program Files\Merticbeat directory

Configuration¶

Editing the file: C:\Program Files\Merticbeat\metricbeat.yml:

In section:

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

change to:

#setup.template.settings:
  #index.number_of_shards: 1
  #index.codec: best_compression

In section:
```
setup.kibana:
```
change to:
```
#setup.kibana:
```

In section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

change to:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

In section:

#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

change to:

output.logstash:
  # The Logstash hosts
  hosts: ["LOGSTASH_IP:5044"]

In section:

#tags: ["service-X", "web-tier"]

change to:

tags: ["metricbeat"]

Run the PowerShell console as Administrator and execute the following commands:

cd 'C:\Program Files\Metricbeat'
.\install-service-metricbeat.ps1

Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Metricbeat\install-service-metricbeat.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R

Output:

Status   Name               DisplayName
------   ----               -----------
Stopped  Metricbeat        Metricbeat

Start Filebeat service:

sc start metricbeat

You can enable, disable and list Metricbeat modules using the following command:

cd 'C:\Program Files\Metricbeat'
metricbeat.exe modules list
metricbeat.exe modules apache enable
metricbeat.exe modules apache disable

Test configuration:

cd 'C:\Program Files\Metricbeat'
metricbeat.exe test config
metricbeat.exe test output

Packetbeat¶

Installation¶

Copy the Packetbeatinstaller from the installation directory install/Agents/beats/windows/packetbeat-oss-6.8.14-windows-x86_64.zip and unpack
Copy the installation files to the C:\Program Files\Packetbeat directory

Configuration¶

Editing the file: C:\Program Files\Packetbeat\packetbeat.yml:

In section:

setup.template.settings:
  index.number_of_shards: 3

change to:

#setup.template.settings:
  #index.number_of_shards: 3

In section:
```
setup.kibana:
```
change to:
```
#setup.kibana:
```

In section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

change to:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

In section:

#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

change to:

output.logstash:
  # The Logstash hosts
  hosts: ["LOGSTASH_IP:5044"]

In section:

#tags: ["service-X", "web-tier"]

change to:

tags: ["packetbeat"]

Run the PowerShell console as Administrator and execute the following commands:

cd 'C:\Program Files\\Packetbeat'
.\install-service-packetbeat.ps1

Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Packetbeat\install-service-packetbeat.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R

Output:

Status   Name               DisplayName
------   ----               -----------
Stopped  Packetbeat        Packetbeat

Start Packetbeat service:

sc start packetbeat

Test configuration:

cd 'C:\Program Files\Packetbeat'
packetbeat.exe test config
packetbeat.exe test output

Linux¶

Filebeat¶

Installation¶

Copy the Filebeat installer from the installation directory install/Agents/beats/linux/filebeat-oss-6.8.14-x86_64.rpm

Install filebeat with following commadn:

yum install -y filebeat-oss-6.8.14-x86_64.rpm

Configuration¶

Editing the file: /etc/filebeat/filebeat.yml:

In section:

- type: log

  # Change to true to enable this input configuration.
  enabled: false

change to:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

In section:

setup.template.settings:
  index.number_of_shards: 1

change to:

#setup.template.settings:
  #index.number_of_shards: 1

In section:
```
setup.kibana:
```
change to:
```
#setup.kibana:
```

In section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

change to:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

In section:

#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

change to:

output.logstash:
  # The Logstash hosts
  hosts: ["LOGSTASH_IP:5044"]

In section:

#tags: ["service-X", "web-tier"]

change to:

tags: ["filebeat"]

Start Filebeat service:

systemctl start filebeat

You can enable, disable and list Filebeat modules using the following command:

filebeat modules list
filebeat modules apache enable
filebeat modules apache disable

Test configuration:

filebeat test config
filebeat test output

Merticbeat¶

Installation¶

Copy the Merticbeatinstaller from the installation directory install/Agents/beats/linux/metricbeat-oss-6.8.14-x86_64.rpm

Install Merticbeat with following command:

yum install -y metricbeat-oss-6.8.14-x86_64.rpm

Configuration¶

Editing the file: /etc/metricbeat/metricbeat.yml:

In section:

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

change to:

#setup.template.settings:
  #index.number_of_shards: 1
  #index.codec: best_compression

In section:
```
setup.kibana:
```
change to:
```
#setup.kibana:
```

In section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

change to:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

In section:

#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

change to:

output.logstash:
  # The Logstash hosts
  hosts: ["LOGSTASH_IP:5044"]

In section:

#tags: ["service-X", "web-tier"]

change to:

tags: ["metricbeat"]

Start Filebeat service:

systemctl start metricbeat

You can enable, disable and list Metricbeat modules using the following command:

metricbeat modules list
metricbeat modules apache enable
metricbeat modules apache disable

Test configuration:

metricbeat test config
metricbeat test output

Packetbeat¶

Installation¶

Copy the Packetbeat installer from the installation directory install/Agents/beats/linux/packetbeat-oss-6.8.14-x86_64.rpm

Install Packetbeatwith following command:

yum install -y packetbeat-oss-6.8.14-x86_64.rpm

Configuration¶

Editing the file: /etc/packetbeat/packetbeat.yml:

In section:

setup.template.settings:
  index.number_of_shards: 3

change to:

#setup.template.settings:
  #index.number_of_shards: 3

In section:
```
setup.kibana:
```
change to:
```
#setup.kibana:
```

In section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

change to:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

In section:

#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

change to:

output.logstash:
  # The Logstash hosts
  hosts: ["LOGSTASH_IP:5044"]

In section:

#tags: ["service-X", "web-tier"]

change to:

tags: ["packetbeat"]

Start Packetbeat service:

servicectl start packetbeat

Test configuration:

packetbeat test config
packetbeat test output