ECS performance with vTM and Graylog

ECS performance with vTM and GraylogECS is usually used with a Load Balancer. Because a load balancer is placed in-band and controls all incoming and outgoing traffic, it can be used to collect performance data and transfer it to external log analyzer for further visualization and alert.

In that post, I’ll share my experience with ECS performance monitoring and reporting based on Brocade vTM load balancer and Graylog log analyzer.

Prepare vTM to collect and send logs

  • Navigate to Virtual Server > Request logging
  • Enable Remote Request logging.
  • Specify IP address of your Graylog server.

ECS performance with vTM and Graylog

  • Specify Custom syslog!format.
%h, %t, %{host}i, %r, %s, %b, %B, %R

Fields used:

  • %h – the client’s IP address
  • %t – time when the last byte was sent to the client
  • %{host}i – the value of a “host” header in the HTTP request
  • %r – first line of the HTTP request
  • %s – status code of HTTP response (e.g. 200)
  • %b – number of bytes sent to the client
  • %B – number of bytes received from the client
  • %R – total time from the first byte of client request until complete response has been sent to client, in seconds

Prepare Graylog syslog message extraction

  • Navigate to System > Inputs > Manage extractors.

ECS performance with vTM and Graylog

  • Specify the Grok patern to extract vTM request logs.
  • Instead of commas (,) in Grok patterns you have to use dots (.)
%{WORD:zeus}: %{IP:srcip}. \[%{HTTPDATE:date}\]. %{IP:dstip}:%{NUMBER:dstport;int}. %{WORD:command} %{URIPATH:path} HTTP/1.1. %{NUMBER:statuscode;int}. %{NUMBER:receivedbytes;int}. %{NUMBER:sentbytes;int}. %{NUMBER:totaltime;float}.

ECS performance with vTM and Graylog

  • Check if example message is extracted correctly.

ECS performance with vTM and Graylog

Create Graylog streams

  • Navigate to Streams > Create Stream.
  • Create new streams. One is for successful READ requests and another for WRITE requests.
  • I created one more stream “bstream” as combined for reads and writes. Actually, that is not necessary.

ECS performance with vTM and Graylog

  • Add stream rules.
  • Stream rule for READ stream is taking into account all GET commands.
  • Stream rule for WRITE stream is based on PUT commands.

ECS perf with vTM and Graylog 6

  • Check the final rules configuration.

ECS performance with vTM and Graylog

  • Start all new streams.

 

Create Graylog pipelines

  • Navigate to System > Configuration.
  • Make sure that Message Processors configuration contains Message Filter Chain before Pipeline Processor. That is needed for correct pipeline rules processing.

ECS performance with vTM and Graylog

  • Navigate to System > Pipelines > Create Rules.
  • Create rules to calculate READ and WRITE bandwidth.
  • totaltime, sentbytes and received bytes values are used (extracted from vTM request logs).

ECS performance with vTM and Graylog

  • READ bandwidth calculation rule.
rule "READ bandwidth"
 when
   has_field("totaltime") && has_field("sentbytes")
 then
     let time = to_double($message.totaltime);
     let size = to_double($message.receivedbytes);
     let bandwidth = size / time;
     let bandwidth_mb = to_double(bandwidth / 1048576.0);
     set_field("read_bandwidth", bandwidth_mb);
     route_to_stream(name: "bstream");
 end

ECS perf with vTM and Graylog 9

  • WRITE bandwidth calculation rule.
rule "WRITE bandwidth"
 when
   has_field("totaltime") && has_field("sentbytes")
 then
     let time = to_double($message.totaltime);
     let size = to_double($message.sentbytes);
     let bandwidth = size / time;
     let bandwidth_mb = to_double(bandwidth / 1048576.0);
     set_field("write_bandwidth", bandwidth_mb);
     route_to_stream(name: "bstream");
 end

ECS performance with vTM and Graylog

  • Navigate to Manage Pipelines.

ECS performance with vTM and Graylog

  • Add new pipelines for READ and WRITE

ECS perf with vTM and Graylog 12

  • Assign corresponding rules to pipelines’ Stage 0.

ECS perf with vTM and Graylog 13

  • Check that rule is assigned to the pipeline stage correctly.

ECS perf with vTM and Graylog 14

  • Navigate to Edit connections.

ECS performance with vTM and Graylog

  • Select READ or WRITE stream to be connected to the pipeline.

ECS performance with vTM and Graylog

Test performance monitoring

  • Navigate to Streams > bstream.

ECS perf with vTM and Graylog 17

  • Nothing is presented because there is no workload yet.

ECS performance with vTM and Graylog

  • Generate workload.
  • I’m using COSbench as workload generator.
  • You can compare COSbench testing results with bandwidth calculated by Graylog. Results should be similar.

ECS performance with vTM and Graylog

  • Navigate to vTM > Activity > View Logs.
  • Check if GET and PUT requests are generated.

ECS performance with vTM and Graylog

  • By default Graylog is presenting data and make statistic for last 5 minutes.

ECS performance with vTM and Graylog

  • I’d like to make short tests. To get average for last 1 minute I need to create new Relative Time Range option which corresponds 1 minute.
  • Click Update button.

ECS performance with vTM and Graylog

  • Create new Timerange: PT1M.

ECS perf with vTM and Graylog 23

  • Navigate to bstream and select the 1 minute timerange.

ECS performance with vTM and Graylog

  • The graphs should start drawing.

ECS performance with vTM and Graylog

  • Type band in the search field.
  • Expand read_bandwidth and write_bandwidth fields.
  • Select Statistics.

ECS performance with vTM and Graylog

  • Statistic presents average, min, max, etc, values for the last 1 minute.
  • Graphs represent workload dynamic.

ECS perf with vTM and Graylog 27

Test alerting

  • Execute a new workload.

ECS performance with vTM and Graylog

  • Average write bandwidth for 5 minutes period is 28.56MB/s.
  • Just for testing purposes let’s configure alerting for bandwidth lower than 30MB/s. So we have to get an alert.

ECS perf with vTM and Graylog 29

  • Navigate to Streams > Create Stream.

ECS perf with vTM and Graylog 30

  • Create a new stream for WRITE medium size objects.

ECS performance with vTM and Graylog

  • Specify two rules:
    • sentbytes > 1000000
    • sentbytes < 100001000

ECS performance with vTM and Graylog

  • Start the stream

ECS perf with vTM and Graylog 33

  • Navigate to Alerts > Manage Conditions.

ECS perf with vTM and Graylog 34

  • Select Add new Condition

ECS performance with vTM and Graylog

  • Specify the stream.
  • Select Field Aggregation Alert

ECS performance with vTM and Graylog

  • Specify alert conditions:
    • Field = write_bandwidth
    • Time range = 5 minutes
    • Threshold type = lower
    • Threshold = 30 (MB/s)
    • Aggregation type = mean value
    • Grace period = 5 minutes
    • Message backlog = 2 messages

ECS performance with vTM and Graylog

  • Check if an alert is generated.

ECS performance with vTM and Graylog

  • Stop a workload and wait for 5 minutes and refresh the screen.
  • There are no unresolved alerts anymore.

ECS performance with vTM and Graylog

  • Click on Show all alerts.
  • Low WRITE bandwidth alert is presented as resolved now.

ECS performance with vTM and Graylog

Note: you can configure sending alerts via email using the Notification feature.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s