slurdge•org

Recently, while working on goeland, I ran some test on reddit’s RSS feeds.

While it worked fine in the past, in the last few months with update to golang 1.17, people started complaining that reddit was rejecting their requests. While if you do the same request with curl, it works, with golang you would get a 429 error code.

The solution outlined on reddit was alike:

req, _ := http.NewRequest("GET", url, nil)
req.Header.Set("User-Agent", "Custom Agent")
var defaultClient = http.Client{
    Transport: &http.Transport{
	TLSNextProto: map[string]func(authority string, c *tls.Conn) http.RoundTripper{},
    },
}
resp, _ := defaultClient.Do(req)

And it worked for a few more months. Alas, these last few days, I was getting 403 exactly like this thread: https://www.reddit.com/r/redditdev/comments/uncu00/go_golang_clients_getting_403_blocked_responses/

I decided enough was enough, time to sort it out. We will fire up wireshark and see what is really happening on the wire. In order to debug this issue, I created a CURL request that was working and replicated it in go with https://mholt.github.io/curl-to-go/ .

However, the traffic is https, so one would need to be able to decrypt it. It’s easy enough with CURL, but with golang you need to have more code in order to do it. I found an excellent article by Ben Burwell that explains how to do it.

package main

import (
  "crypto/tls"
  "net/http"
  "os"
)

func main() {
  f, err := os.OpenFile("/tmp/keys", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600)
  if err != nil {
    panic(err)
  }
  defer f.Close()

  client := &http.Client{
    Transport: &http.Transport{
      TLSClientConfig: &tls.Config{
        KeyLogWriter: f,
      },
    },
  }

  client.Get("https://www.benburwell.com")
}

But as soon I put this in place, the 429/403 error dissapeared! Actually, the ‘fix’ is even simpler, you just need to have an empty (default, zero value) TLSClientConfig like so:

defaultClient = http.Client{
	Transport: &http.Transport{
		TLSClientConfig: &tls.Config{
		},
	},
}

So the problem is fixed, however it becomes impossible to debug, as debugging it removes the reddit bug… Even by looking at the different between http.DefaultClient and the one created, nothing major catches the eye.

Let’s hope this fix will work longer…