Creating a Lark Bot to Send Notifications When Errors Occur in Production

Husni Nur Fadillah
Husni Nur Fadillah • • 4 min read
Creating a Lark Bot to Send Notifications When Errors Occur in Production

Background

Errors in production must be addressed quickly. To do this, you need a bot that can notify developers when errors occur. This bot automates message delivery to specific groups within your team.

Since my company uses Lark as its messaging application, I'll create the bot in Lark.

Implementation

When an error occurs while a user interacts with the application, we first check if it's the same error that occurred a few seconds ago. If it's different, we set a key with its value and an expiration time. Then, we send a notification to Lark. However, if it's the same error, the bot won't send a notification.

Setting an expiration time on the Redis key implements rate limiting. The goal is to prevent the bot from sending repeated messages with the same information.

1. Create a Custom Bot in a Lark Group

Open the settings of the group where you want to add the bot, then select BOTs. Note that you can create more than one bot in a single group.

Fill out the form provided, then copy the Webhook URL that's given. This webhook allows you to integrate the application you create with Lark, enabling you to automate activities or processes in Lark.

2. Error Alert Notifications in a Golang Project

I'll use the project from my previous blog post, which already has a logger installed. The additional dependency is go-lark.

In this article, I'll create notifications that appear when the error status is 4xx or 5xx. But first, let me create the message template. The CardMessageTemplate function will render the message in a format that Lark can read.

func SendNotifLark(errMsg string, code int, reqData string, respData ResponseData) {
 url := Config("LARK_WEBHOOK_URL")
 bot := lark.NewNotificationBot(url)

 cardData := CardMsgData{
  errMsg:       errMsg,
  hyperLinkBtn: Config("SEQ_URL"),
  reqData:      reqData,
  respData:     respData,
 }

 cardCst := CardMessageTemplate(cardData)
 msg := lark.NewMsgBuffer(lark.MsgInteractive)
 msg.Card(cardCst.String())

 bot.PostNotificationV2(msg.Build())
}
 
You can customize the message template however you like. The go-lark repository provides several examples.

Go Lark Examples

Next, create a function to send the notification:

func SendNotifLark(errMsg string, code int, reqData string, respData ResponseData) {
 url := Config("LARK_WEBHOOK_URL")
 bot := lark.NewNotificationBot(url)

 cardData := CardMsgData{
  errMsg:       errMsg,
  hyperLinkBtn: Config("SEQ_URL"),
  reqData:      reqData,
  respData:     respData,
 }

 cardCst := CardMessageTemplate(cardData)
 msg := lark.NewMsgBuffer(lark.MsgInteractive)
 msg.Card(cardCst.String())

 bot.PostNotificationV2(msg.Build())
}
 

3. Prevent Message Spam

As explained earlier, I'll implement rate limiting to prevent message spam using Redis caching.

func SendNotifLark(rateLimit int, rateLimitKey string, errMsg string, code int, reqData string, respData ResponseData) {
 url := Config("LARK_WEBHOOK_URL")
 bot := lark.NewNotificationBot(url)

 cardData := CardMsgData{
  errMsg:       errMsg,
  hyperLinkBtn: Config("SEQ_URL"),
  reqData:      reqData,
  respData:     respData,
 }

 cardCst := CardMessageTemplate(cardData)

 msg := lark.NewMsgBuffer(lark.MsgInteractive)
 msg.Card(cardCst.String())

 if rateLimit > 0 {
  // Use a default rateLimitKey if not provided
  if rateLimitKey == "" {
   rateLimitKey = errMsg
  }

  cacheExist, err := RDB.SetNX(context.Background(), rateLimitKey, 1, time.Duration(rateLimit)*time.Second).Result()
  if err != nil {
   // Handle Redis errors here
  }

  if cacheExist {
   bot.PostNotificationV2(msg.Build())
  }
 } else {
  // If rateLimit is not greater than 0, post the notification immediately
  bot.PostNotificationV2(msg.Build())
 }
}

Here I've added the option to implement rate limiting or not. The SetNX function is used to set a key in Redis with the following options:

  1. NX (No Exist): This function uses the NX option, meaning the key will only be set if it doesn't already exist in the Redis database. In other words, if the key already exists, the operation won't be performed.
  2. EX (Expire): This function also uses the EX (expire) option to set an expiration time on the newly created key. The value given for EX specifies how many seconds the key will remain before being automatically deleted from the Redis database.

Alert Notifications with Seq

You can do this if you don't need to perform customization by leveraging the Alert feature that Seq provides.

You can explore this yourself, but what you should note here is that the plugin or Output app instance used depends on the messaging app you're using. For Slack, use Seq.App.Slack; for Microsoft Teams, use Seq.App.Teams, and so on.

I'm using Lark here, but Seq doesn't provide a plugin for Lark. However, we can still use Seq.App.HttpRequest. After that, add an instance using that plugin.

Fill in the URL with the webhook from the Custom Bot Lark created earlier and change the Method to POST. Customize the message that this alert will send in the Body field.

Closing

That concludes the article "Creating a Lark Bot to Send Notifications When Errors Occur in Production." If you'd like to discuss or provide feedback, please leave a comment below.

I hope you enjoyed reading this article and found what you were looking for.

Share this post