Examples

Create frequency graph from a log

Task: Read /var/log/dpkg.log and create a graph to visualize how often packages are installed, upgraded and removed.

Solution: The loop (34) calls function read_log which reads the log line by line (13), splits the fields (14) and concatenate date l[0] and time l[1] in minutes (15). Third field of the log l[2] is status of the dpkg operation(install, upgrade, remove …). Method zincrby (16) increments by 1 the score of word in the key l[2]. As a result the database contains keys(install, upgrade, remove …) and associated lists of words sorted by score. Next loop (38) calls the function write_csv with all keys. As a result status.csv files are created in the current directory with the (date;score) pairs.

[create-graph-01.rb]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/ruby
# Tested with ruby 2.3.3, ruby-redis 3.3.3 and redis 4.0.1

require 'redis'

LOG_FILES = ['/var/log/dpkg.log', ]
LOG_SEPARATOR = ' '
CSV_SEPARATOR = ';'

def read_log(log_file, r)
  # This function reads log_file and put the status into the database
  f = File.open(log_file, "r")
  f.each do |line|
    l = line.split(LOG_SEPARATOR)
    word = l[0] + " " + l[1][0..-4]
    r.zincrby(l[2], 1, word)
  end
  f.close
end

def write_csv(status, r)
  # This function reads the database and writes the status CSV file
  f = File.open(status + ".csv", "w")
  l = r.zrange(status, 0, -1, :with_scores => true)
  l.each do |x|
    f.write( x[0] + CSV_SEPARATOR + x[1].to_s + "\n")
  end
  f.close
end

r = Redis.new(host: "localhost", port: 6379, db: 0)
r.flushdb

for log_file in LOG_FILES
  read_log(log_file, r)
end

for status in r.keys
  write_csv(status, r)
end

Result: The status.csv files can be used to create a graph with gnuplot.

[create-graph-01.gnuplot]

_images/graph-01.png

List 10 most used words in a text

Task: Read text from a file and list 10 most frequently used words in it.

Solution: Let’s use article about Redis at wikipedia.org as a text.

[create-topchart-text.bash]

#!/bin/bash
lynx -dump -nolist https://en.wikipedia.org/wiki/Redis > redis.txt

zincrby (12) increments by 1 the score of word in the key topchart and zrange (14) returns top 10 words with scores.

[create-topchart.rb]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
#!/usr/bin/ruby
# Tested with ruby 2.3.3, ruby-redis 3.3.3 and redis 4.0.1

require 'redis'

FILE = 'redis.txt'

r = Redis.new(host: "localhost", port: 6379, db: 0)
r.flushdb

f = File.open(FILE, "r")
f.read.scan(/\w+/).each { |word| r.zincrby("topchart", 1, word)}

ranking = r.zrange("topchart", -10, -1, :with_scores => true)

for x in ranking
  puts ( x[1].to_s + " " + x[0] )
end

Result:

> ./create-topchart.rb
11.0 Retrieved
13.0 edit
23.0 in
24.0 a
24.0 is
26.0 and
33.0 of
34.0 to
37.0 the
69.0 Redis