Here's an example of how you can web scrape Amazon.com using the Go programming language and the GoQuery library:
First, install the GoQuery library by running go get github.com/PuerkitoBio/goquery in your command line.
Next, import the necessary libraries:
package main
import (
"fmt"
"github.com/PuerkitoBio/goquery"
"net/http"
)
Use the http.Get() function to make a GET request to the Amazon website. For example:
response, err := http.Get("https://www.amazon.com")
Verify that the request was successful by checking the err variable. If it is nil, the request was successful:
if err != nil {
fmt.Println("Request failed:", err)
return
}
Use the goquery.NewDocumentFromResponse() function to create a goquery document from the response:
doc, err := goquery.NewDocumentFromResponse(response)
Use the goquery document to find the specific elements you want to scrape using the Selection.Find() or Selection.Find() methods.
For example, to find all of the product links on the page:
doc.Find("a.a-link-normal.s-no-outline").Each(func(i int, s *goquery.Selection) {
fmt.Println(s.Text())
})
Extract the information you want using the Selection.Text() or Selection.Attr() methods.
doc.Find("span.a-price-whole").Each(func(i int, s *goquery.Selection) {
price := s.Text()
fmt.Println("Price:", price)
})
Remember to be respectful and to comply with website's TOS, also you may need to use sleep functions or rotate IP's or headers to avoid IP blocking.
That's it! you have a basic understanding of how to scrape a website using Go and GoQuery. With a little more practice and exploration of the library's various methods and attributes, you'll be able to extract much more information from a website.
Please keep in mind that web scraping can be against website's TOS and it's important to check the website's policy before scraping the website.
