Feature #11880

Centralize our servers' logs

Added by intrigeri 2016-10-16 16:45:51 . Updated 2019-04-07 09:25:54 .

Status:
Confirmed
Priority:
Low
Assignee:
Category:
Infrastructure
Target version:
Start date:
2016-10-16
Due date:
% Done:

0%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Right now we have volatile Journal + some persistent log files managed by rsyslog and individual applications. This sometimes makes it painful to debug problems since one has to cross-match info from various sources. It would be nicer if all our logs landed in a single place.

An initial idea to start brainstorming about it would be:

  • on each of our systems, send all logs to journald
  • configure these journalds to have volatile storage only (that’s the default and what we currently do) and to forward them to a single journald instance running in a central place
  • in the central logging location, either have journald store logs in a persistent manner, or forward them to a fancy system like Graylog (that seems much easier to setup than an ELK stack)

Subtasks

History

#1 Updated by intrigeri 2016-10-16 16:47:11

  • Assignee set to bertagaz
  • QA Check set to Info Needed

bertagaz, does the rationale make sense to you? What about the initial implementation idea? (You can ignore the Graylog part for now if you want, it can be added later and thus considered independently.)

#2 Updated by bertagaz 2016-10-19 11:26:16

  • Assignee changed from bertagaz to intrigeri

intrigeri wrote:
> bertagaz, does the rationale make sense to you?

It does!

> What about the initial implementation idea? (You can ignore the Graylog part for now if you want, it can be added later and thus considered independently.)

Sounds reasonable. It raises the question about where to host this centralized log server though. If we were just storing the logs, maybe centralizing on the host would be enough. But Graylog (or similar) could be nice, and then we can’t host that on the host itself. I also didn’t look if it was packaged in Debian.

Did you open that because you intended to work on that?

#3 Updated by intrigeri 2016-10-19 13:38:08

> It raises the question about where to host this centralized log server though. If we were just storing the logs, maybe centralizing on the host would be enough. But Graylog (or similar) could be nice, and then we can’t host that on the host itself.

I had in a mind a dedicated VM on lizard.

> I also didn’t look if it was packaged in Debian.

graylog is not packaged, but most of its dependencies are, and there’s a Puppet module to set it up and manage it.

> Did you open that because you intended to work on that?

I’d like to do the centralized logging part soonish. The Graylog thing is something else entirely, and I don’t want to bother about it too much before we are centralizing our logs already.

#4 Updated by intrigeri 2016-10-26 14:29:12

  • Subject changed from Consider centralizing our servers' logs to Centralize our servers' logs
  • QA Check deleted (Info Needed)

#5 Updated by intrigeri 2017-06-05 17:18:22

  • Priority changed from Normal to Low

#6 Updated by intrigeri 2019-04-07 09:25:54

  • Assignee deleted (intrigeri)