Log rotation in ViPR

Logs rotationLet’s check how ViPR Controller takes care about capacity consumed by log files.

The default location for logs is /opt/storageos/logs . That is the hard link to /var/log.

vipr1:~ # ls /opt/storageos/logs/
ConsoleKit dbsvc.log.20141203-194829.gz objcontrolsvc.out
YaST2 dbsvc.out portalsvc.log
apache2 fabricctrlsvc.log portalsvc.out
apisvc.log fabricctrlsvc.out sa
apisvc.out firewall sasvc.err
atop genconfig.log sasvc.err.old
audit geodbsvc.log sasvc.log
auth geodbsvc.out sasvc.out
auth.checkpoint geosvc.log sasvc.out.old
authsvc.log geosvc.out storageserver.log
authsvc.out keepalived.log syssvc.log
boot.msg krb5 syssvc.out
boot.omsg lastlog systemevents
btmp localmessages tallylog
connectemc.log messages vasasvc.log
controllersvc.log nginx.log vasasvc.out
controllersvc.log.20141201-160246.gz nginx_access.log warn
controllersvc.out nginx_error.log wtmp
coordinatorsvc.log nodetool.log zkutils.log
coordinatorsvc.out ntp zypp
dbsvc.log objcontrolsvc.log
dbsvc.log.20141203-163649.gz objcontrolsvc.log.20141130-132448.gz

Most of ViPR related logs are not deleted, but rotated by Log4j.
Config files for different components are in /opt/storageos/conf/*log4j.properties files

vipr1:~ # ls /opt/storageos/conf/*log4j.properties
 /opt/storageos/conf/apisvc-log4j.properties
 /opt/storageos/conf/authsvc-log4j.properties
 /opt/storageos/conf/bkutils-log4j.properties
 /opt/storageos/conf/blobsvc-junit-log4j.properties
 /opt/storageos/conf/blobsvc-log4j.properties
 /opt/storageos/conf/cassandra-cli-log4j.properties
 /opt/storageos/conf/cm-junit-log4j.properties
 /opt/storageos/conf/cm-log4j.properties
 /opt/storageos/conf/controllersvc-log4j.properties
 /opt/storageos/conf/coordinatorsvc-log4j.properties
 /opt/storageos/conf/dbchecker-log4j.properties
 /opt/storageos/conf/dbcli-log4j.properties
 /opt/storageos/conf/dbnegativetest-log4j.properties
 /opt/storageos/conf/dbsvc-log4j.properties
 /opt/storageos/conf/dbtest-log4j.properties
 /opt/storageos/conf/dbutils-log4j.properties
 /opt/storageos/conf/downloadiso-log4j.properties
 /opt/storageos/conf/dtquery-junit-log4j.properties
 /opt/storageos/conf/dtquery-log4j.properties
 /opt/storageos/conf/fabricctrlsvc-log4j.properties
 /opt/storageos/conf/geodbsvc-log4j.properties
 /opt/storageos/conf/geodbtest-log4j.properties
 /opt/storageos/conf/georeceiver-junit-log4j.properties
 /opt/storageos/conf/georeceiver-log4j.properties
 /opt/storageos/conf/georeplayer-junit-log4j.properties
 /opt/storageos/conf/georeplayer-log4j.properties
 /opt/storageos/conf/geosvc-log4j.properties
 /opt/storageos/conf/geosvctest-log4j.properties
 /opt/storageos/conf/hdfssvc-log4j.properties
 /opt/storageos/conf/log4j.properties
 /opt/storageos/conf/nodetool-log4j.properties
 /opt/storageos/conf/objcontrolsvc-log4j.properties
 /opt/storageos/conf/objheadsvc-log4j.properties
 /opt/storageos/conf/portalsvc-log4j.properties
 /opt/storageos/conf/rm-junit-log4j.properties
 /opt/storageos/conf/rm-log4j.properties
 /opt/storageos/conf/sasvc-log4j.properties
 /opt/storageos/conf/scenariotest-log4j.properties
 /opt/storageos/conf/simulators-log4j.properties
 /opt/storageos/conf/ssm-junit-log4j.properties
 /opt/storageos/conf/ssm-log4j.properties
 /opt/storageos/conf/syssvc-log4j.properties
 /opt/storageos/conf/vasasvc-log4j.properties
 /opt/storageos/conf/zkutils-log4j.properties

By default, logs are gzipped after 50MB size.

vipr1:~ # cat /opt/storageos/conf/dbsvc-log4j.properties
 # output messages into a rolling log file
 log4j.rootLogger=INFO,R

# set at WARN to filter out user level exception msgs
 log4j.logger.org.apache.zookeeper=WARN

# rolling log file
 log4j.appender.R=org.apache.log4j.rolling.RollingFileAppender
 log4j.appender.R.layout=org.apache.log4j.PatternLayout
 log4j.appender.R.layout.ConversionPattern=%d{ISO8601} [%t] %5p %F (line %L) %m%n

# triggering policy
 log4j.appender.R.TriggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
 # max file size is set to 50MB
 log4j.appender.R.TriggeringPolicy.MaxFileSize=52428800

# rolling policy
 log4j.appender.R.RollingPolicy=com.emc.storageos.management.jmx.logging.TimeWindowRollingPolicy
 log4j.appender.R.RollingPolicy.ActiveFileName=${product.home}/logs/dbsvc.log
 log4j.appender.R.RollingPolicy.FileNamePattern=${product.home}/logs/dbsvc.log.%d{yyyyMMdd-HHmmss}.gz

Option that limits the number of logs’ archives (log4j.appender.R.MaxBackupIndex) is not used.

Logs are purged from the system and we retain 7 days. So any collection spanning farther back would include the current weeks log sets only.

vipr1:~ # cat /etc/cron.hourly/cleanup_logs

...
 LOG_FILE_PATTERN="*.log*.gz"

RESERVE_DAYS=7
 CHECK_INTERVAL=100
 ...

Change the default duration of 7 days to something longer is not really recommended because there would be added overhead for the UI parser to present live ViPR logging details a longer time frame. So if we make such a decision we’d need reach balance to retain UI performance of live log feeds/parsing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s