Compare commits

..

25 Commits

Author SHA1 Message Date
T.v.Dein
239e253057 Make ad directory name tunable, adapt to kleinanzeigen.de site changes
Add template for ad name, adapt kleinanzeigen.de changes
2024-01-12 14:56:08 +01:00
cdf58efd45 fixed changes on kleinanzeigen.de:
- Meta did not contain condition and category together anymore, they
removed  the category. Therefore fetching (that is, validation)
failed.
- Now we extract the condition and category directly.
- On top, category now includes the whole category tree.
- unit tests had to be tweaked for this measure.
2024-01-12 14:11:02 +01:00
110ee17091 fixed utf8 2024-01-12 13:31:24 +01:00
8321d3c343 added template for ad directory, by default include id now 2024-01-12 13:29:59 +01:00
T.v.Dein
56f53bb777 remove duplicate license badge (#28)
* remove duplicate license badge

* fix badges
2024-01-06 18:07:33 +01:00
T.v.Dein
9e7f9a2821 Merge pull request #27 from TLINDEN/test/add-main-tests
Test/add main tests
2024-01-02 12:51:51 +01:00
577f9d983e portable error check 2024-01-02 12:36:00 +01:00
114f6b16d9 also added coverage report+badge 2024-01-02 12:31:19 +01:00
a06c730fe4 put all tests into main_test.go, more failure mode tests and verify 2024-01-02 12:22:52 +01:00
d8e968ed6d better error message on 404 2024-01-02 12:22:26 +01:00
5f450e54ea add commandline main() test units 2024-01-01 20:53:39 +01:00
3e6349cf36 pass a io.Writer to loggers and outputs so we can test the cmdline 2024-01-01 20:53:05 +01:00
T.v.Dein
bf8e074034 Merge pull request #26 from TLINDEN/test/enhancements
Enhanced error checking, added more failure tests
2024-01-01 16:27:20 +01:00
6dea8d78ed added more invalid tests 2024-01-01 16:24:43 +01:00
ea76b98445 upd httpmock+deps 2024-01-01 16:24:33 +01:00
9f688b7692 put ad code into separate file, enhance error checking 2024-01-01 16:24:07 +01:00
T.v.Dein
d8baa34c54 Test/add mock tests (#24)
* add scrape unit test using httpmock lib
2023-12-29 13:47:18 +01:00
T.v.Dein
c1cbce32e1 fix linter errors (#23) 2023-12-23 22:51:50 +01:00
T.v.Dein
dc4d3d7f9c add ci pipeline (#22)
Co-authored-by: Thomas von Dein <tom@izb.net>
2023-12-23 22:36:21 +01:00
T.v.Dein
b28f544416 Doc/add prior art (#21)
* add mor prior art
2023-12-23 22:20:39 +01:00
T.v.Dein
8cdefe457b added windows screenshots (#20)
Co-authored-by: Thomas von Dein <tom@vondein.org>
2023-12-23 18:48:10 +01:00
T.v.Dein
1e4b406aa4 Revert "Fix/newline windows (#18)" (#19)
This reverts commit eaf4db6cef.
2023-12-23 18:00:51 +01:00
T.v.Dein
eaf4db6cef Fix/newline windows (#18)
* fix #17: use fmt.Println() after stats
* bump version
2023-12-22 18:59:08 +01:00
T.v.Dein
825649bb3b added screenshots and a section about prior work (#16) 2023-12-21 12:25:04 +01:00
T.v.Dein
6aa9c658b6 add doc link (#15)
Co-authored-by: Thomas von Dein <tom@vondein.org>
2023-12-19 18:36:11 +01:00
27 changed files with 873 additions and 78 deletions

BIN
.github/assets/adlisting-windows.jpg vendored Executable file

Binary file not shown.

After

Width:  |  Height:  |  Size: 139 KiB

BIN
.github/assets/cmd-windows.jpg vendored Executable file

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

BIN
.github/assets/kleinanzeigen-ad.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 199 KiB

BIN
.github/assets/kleinanzeigen-backup.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 263 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

BIN
.github/assets/kleinanzeigen-index.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 232 KiB

BIN
.github/assets/liste-windows.jpg vendored Executable file

Binary file not shown.

After

Width:  |  Height:  |  Size: 90 KiB

47
.github/workflows/ci.yaml vendored Normal file
View File

@@ -0,0 +1,47 @@
name: build-and-test
on: [push, pull_request]
jobs:
build:
strategy:
matrix:
version: [1.21]
os: [ubuntu-latest, windows-latest, macos-latest]
name: Build
runs-on: ${{ matrix.os }}
steps:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: ${{ matrix.version }}
id: go
- name: checkout
uses: actions/checkout@v3
- name: build
run: go build
- name: test
run: make test
- name: Update coverage report
uses: ncruces/go-coverage-report@main
with:
report: true
chart: true
amend: true
if: |
matrix.os == 'ubuntu-latest' &&
github.event_name == 'push'
continue-on-error: true
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- uses: actions/setup-go@v3
with:
go-version: 1.21
- uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3

1
.gitignore vendored
View File

@@ -1,3 +1,4 @@
test
kleingebaeck
releases
t/out

View File

@@ -50,9 +50,10 @@ install: buildlocal
install -o $(UID) -g $(GID) -m 444 $(tool).1 $(PREFIX)/man/man1/
clean:
rm -rf $(tool) coverage.out testdata
rm -rf $(tool) coverage.out testdata t/out
test: clean
mkdir -p t/out
go test ./... $(ARGS)
testfuzzy: clean

View File

@@ -2,8 +2,9 @@
![Kleingebaeck Logo](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/kleingebaecklogo-small.png)
[![License](https://img.shields.io/badge/license-GPL-blue.svg)](https://github.com/tlinden/kleingebaeck/blob/master/LICENSE)
[![Go Report Card](https://goreportcard.com/badge/github.com/tlinden/kleingebaeck)](https://goreportcard.com/report/github.com/tlinden/kleingebaeck)
[![Actions](https://github.com/tlinden/kleingebaeck/actions/workflows/ci.yaml/badge.svg)](https://github.com/tlinden/kleingebaeck/actions)
[![Go Coverage](https://github.com/tlinden/kleingebaeck/wiki/coverage.svg)](https://raw.githack.com/wiki/tlinden/kleingebaeck/coverage.html)
![GitHub License](https://img.shields.io/github/license/tlinden/kleingebaeck)
[![GitHub release](https://img.shields.io/github/v/release/tlinden/kleingebaeck?color=%2300a719)](https://github.com/TLINDEN/kleingebaeck/releases/latest)
@@ -15,6 +16,38 @@ directory, each ad into its own subdirectory. The backup will contain
a textfile `Adlisting.txt` which contains the ad contents as the
title, body, price etc. All images will be downloaded as well.
## Screenshots
This is the index of my kleinanzeigen.de Account:
![Index](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/kleinanzeigen-index.png)
Here I download my ads on the commandline:
![Download](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/kleinanzeigen-download.png)
And this is the backup directory after download:
![Download](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/kleinanzeigen-backup.png)
Here's a directory for one ad:
![Download](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/kleinanzeigen-ad.png)
**The same thing under windows:**
Downloading ads:
![Download](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/cmd-windows.jpg)
Backup directory after download:
![Download](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/liste-windows.jpg)
And one ad listing directory:
![Download](https://github.com/TLINDEN/kleingebaeck/blob/main/.github/assets/adlisting-windows.jpg)
## Installation
The tool doesn't need authentication and doesn't have any
@@ -128,6 +161,10 @@ variable. The supplied sample config contains the default template.
All images will be stored in the same directory.
## Documentation
You can read the documentation [online](https://github.com/TLINDEN/kleingebaeck/blob/main/kleingebaeck.pod) or locally once you have installed kleingebaeck with: `kleingebaeck --manual`.
## Kleingebäck?
The name is derived from "kleinanzeigen backup": "klein" (german for
@@ -147,6 +184,34 @@ https://github.com/TLINDEN/kleingebaeck/issues.
Please repeat the failing command with debugging enabled `-d` and
include the output in the issue.
## Related projects
I could not find any projects specifically designed to backup
kleinanzeigen.de ads, however there's a bot project which is also able
to download ads:
[kleinanzeigen-bot](https://github.com/Second-Hand-Friends/kleinanzeigen-bot/). However,
be aware that kleinanzeigen.de is actively fighting bots! Look at this
[issue](https://github.com/Second-Hand-Friends/kleinanzeigen-bot/issues/219). The
problem with these kind of bots is, that they login into your account
using your credentials. If the company is able to detect bot activity
they can associate it easily with your account and **lock you
out**. So be careful.
**kleingebäck** doesn't need to login, it just accesses public
available web pages. Kleinanzeigen.de could hardly do anything against
it, once because it is legal. There's no difference between a browser
and a commandline client. Both run on the clientside and it is not
kleinanzeigen.de's decision which software one uses to access their
pages. And second: because you can use it to download any ads, not
just yours. So it is not really clear if the activity is associated in
any way with the ad owner. In addition to that comes the fact that
kleingebäck is just a backup tool. It is not intendet to be used on a
daily basis. You cannot use it to view regular ads or maintain your
own ads. You'll need to use the mobile app or the browser page with a
login. So, in my point of view, the risk is very minimal.
There is another Tool available named [kleinanzeigen-enhanded](https://kleinanzeigen-enhanced.de/). It is a complete Ad management system targeting primarily commercial users. You have to pay a monthly fee, perhaps there's also a free version available, but I haven't checked. The tool is implemented as a Chrome browser extension, which explains why it was possible to implement it without an API. It seems to be a nice solution for power users by the looks of it. And it includes backups.
## Copyright und License
Licensed under the GNU GENERAL PUBLIC LICENSE version 3.

69
ad.go Normal file
View File

@@ -0,0 +1,69 @@
/*
Copyright © 2023 Thomas von Dein
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
package main
import (
"log/slog"
"strings"
)
type Index struct {
Links []string `goquery:".text-module-begin a,[href]"`
}
type Ad struct {
Title string `goquery:"h1"`
Slug string
Id string
Condition string `goquery:".addetailslist--detail--value,text"`
Category string
CategoryTree []string `goquery:".breadcrump-link,text"`
Price string `goquery:"h2#viewad-price"`
Created string `goquery:"#viewad-extra-info,text"`
Text string `goquery:"p#viewad-description-text,html"`
Images []string `goquery:".galleryimage-element img,[src]"`
}
// Used by slog to pretty print an ad
func (ad *Ad) LogValue() slog.Value {
return slog.GroupValue(
slog.String("title", ad.Title),
slog.String("price", ad.Price),
slog.String("id", ad.Id),
slog.Int("imagecount", len(ad.Images)),
slog.Int("bodysize", len(ad.Text)),
slog.String("categorytree", strings.Join(ad.CategoryTree, "+")),
slog.String("condition", ad.Condition),
)
}
// check for completeness. I erected these fields to be mandatory
// (though I really don't know if they really are). I consider images
// and meta optional. So, if either of the checked fields here is
// empty we return an error. All the checked fields are extracted
// using goquery. However, I think price is optional since there are
// ads for gifts as well.
//
// Note: we return true for "ad is incomplete" and false for "ad is complete"!
func (ad *Ad) Incomplete() bool {
if ad.Category == "" || ad.Created == "" || ad.Text == "" {
return true
}
return false
}

View File

@@ -1,5 +1,5 @@
/*
Copyright © 2023 Thomas von Dein
Copyright © 2023-2024 Thomas von Dein
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -19,6 +19,7 @@ package main
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"runtime"
@@ -32,7 +33,7 @@ import (
)
const (
VERSION string = "0.1.0"
VERSION string = "0.1.1"
Baseuri string = "https://www.kleinanzeigen.de"
Listuri string = "/s-bestandsliste.html"
Defaultdir string = "."
@@ -42,6 +43,7 @@ const (
"Category: {{.Category}}\r\nCondition: {{.Condition}}\r\nCreated: {{.Created}}\r\n\r\n{{.Text}}\r\n"
Useragent string = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " +
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
DefaultAdNameTemplate string = "{{.Slug}}-{{.Id}}"
)
const Usage string = `This is kleingebaeck, the kleinanzeigen.de backup tool.
@@ -71,6 +73,7 @@ type Config struct {
User int `koanf:"user"`
Outdir string `koanf:"outdir"`
Template string `koanf:"template"`
Adnametemplate string `koanf:"adnametemplate"`
Loglevel string `koanf:"loglevel"`
Limit int `koanf:"limit"`
Adlinks []string
@@ -87,7 +90,7 @@ func (c *Config) IncrImgs(num int) {
}
// load commandline flags and config file
func InitConfig() (*Config, error) {
func InitConfig(w io.Writer) (*Config, error) {
var k = koanf.New(".")
// determine template based on os
@@ -97,17 +100,20 @@ func InitConfig() (*Config, error) {
}
// Load default values using the confmap provider.
k.Load(confmap.Provider(map[string]interface{}{
"template": template,
"outdir": ".",
"loglevel": "notice",
"userid": 0,
}, "."), nil)
if err := k.Load(confmap.Provider(map[string]interface{}{
"template": template,
"outdir": ".",
"loglevel": "notice",
"userid": 0,
"adnametemplate": DefaultAdNameTemplate,
}, "."), nil); err != nil {
return nil, err
}
// setup custom usage
f := flag.NewFlagSet("config", flag.ContinueOnError)
f.Usage = func() {
fmt.Println(Usage)
fmt.Fprintln(w, Usage)
os.Exit(0)
}
@@ -122,7 +128,9 @@ func InitConfig() (*Config, error) {
f.BoolP("help", "h", false, "show usage")
f.BoolP("manual", "m", false, "show manual")
f.Parse(os.Args[1:])
if err := f.Parse(os.Args[1:]); err != nil {
return nil, err
}
// generate a list of config files to try to load, including the
// one provided via -c, if any

5
go.mod
View File

@@ -4,6 +4,7 @@ go 1.21
require (
astuart.co/goq v1.0.0
github.com/jarcoal/httpmock v1.3.1
github.com/knadh/koanf/parsers/toml v0.1.0
github.com/knadh/koanf/providers/confmap v0.1.0
github.com/knadh/koanf/providers/file v0.1.0
@@ -12,6 +13,7 @@ require (
github.com/lmittmann/tint v1.0.3
github.com/mattn/go-isatty v0.0.20
github.com/spf13/pflag v1.0.5
golang.org/x/sync v0.5.0
)
require (
@@ -23,8 +25,7 @@ require (
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
golang.org/x/net v0.0.0-20190606173856-1492cefac77f // indirect
golang.org/x/sync v0.5.0 // indirect
golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
golang.org/x/sys v0.6.0 // indirect
)

7
go.sum
View File

@@ -9,6 +9,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/jarcoal/httpmock v1.3.1 h1:iUx3whfZWVf3jT01hQTO/Eo5sAYtB2/rqaUuOtpInww=
github.com/jarcoal/httpmock v1.3.1/go.mod h1:3yb8rc4BI7TCBhFY8ng0gjuLKJNquuDNiPaZjnENuYg=
github.com/knadh/koanf/maps v0.1.1 h1:G5TjmUh2D7G2YWf5SQQqSiHRJEjaicvU0KpypqB3NIs=
github.com/knadh/koanf/maps v0.1.1/go.mod h1:npD/QZY3V6ghQDdcQzl1W4ICNVTkohC8E73eI2xW4yI=
github.com/knadh/koanf/parsers/toml v0.1.0 h1:S2hLqS4TgWZYj4/7mI5m1CQQcWurxUz6ODgOub/6LCI=
@@ -25,6 +27,8 @@ github.com/lmittmann/tint v1.0.3 h1:W5PHeA2D8bBJVvabNfQD/XW9HPLZK1XoPZH0cq8NouQ=
github.com/lmittmann/tint v1.0.3/go.mod h1:HIS3gSy7qNwGCj+5oRjAutErFBl4BzdQP6cJZ0NfMwE=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/maxatome/go-testdeep v1.12.0 h1:Ql7Go8Tg0C1D/uMMX59LAoYK7LffeJQ6X2T04nTH68g=
github.com/maxatome/go-testdeep v1.12.0/go.mod h1:lPZc/HAcJMP92l7yI6TRz1aZN5URwUBUAfUNvrclaNM=
github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
@@ -44,8 +48,9 @@ github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190606173856-1492cefac77f h1:IWHgpgFqnL5AhBUBZSgBdjl2vkQUEzcY+JNKWfcgAU0=
golang.org/x/net v0.0.0-20190606173856-1492cefac77f/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/sync v0.5.0 h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE=
golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=

View File

@@ -133,7 +133,7 @@
.\" ========================================================================
.\"
.IX Title "KLEINGEBAECK 1"
.TH KLEINGEBAECK 1 "2023-12-19" "1" "User Commands"
.TH KLEINGEBAECK 1 "2024-01-12" "1" "User Commands"
.\" For nroff, turn off justification. Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
@@ -142,7 +142,7 @@
kleingebaeck \- kleinanzeigen.de backup tool
.SH "SYNOPSYS"
.IX Header "SYNOPSYS"
.Vb 10
.Vb 11
\& Usage: kleingebaeck [\-dvVhmoc] [<ad\-listing\-url>,...]
\& Options:
\& \-\-user \-u <uid> Backup ads from user with uid <uid>.
@@ -153,6 +153,7 @@ kleingebaeck \- kleinanzeigen.de backup tool
\& \-\-config \-c <file> Use config file <file> (default: ~/.kleingebaeck).
\& \-\-manual \-m Show manual.
\& \-\-help \-h Show usage.
\& \-\-version \-V Show program version.
.Ve
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
@@ -235,7 +236,20 @@ Also there's currently no parallelization implemented. This will
change in the future.
.SH "LICENSE"
.IX Header "LICENSE"
Licensed under the \s-1GNU GENERAL PUBLIC LICENSE\s0 version 3.
Copyright 2023\-2024 Thomas von Dein
.PP
This program is free software: you can redistribute it and/or modify
it under the terms of the \s-1GNU\s0 General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
.PP
This program is distributed in the hope that it will be useful,
but \s-1WITHOUT ANY WARRANTY\s0; without even the implied warranty of
\&\s-1MERCHANTABILITY\s0 or \s-1FITNESS FOR A PARTICULAR PURPOSE.\s0 See the
\&\s-1GNU\s0 General Public License for more details.
.PP
You should have received a copy of the \s-1GNU\s0 General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
.SH "Author"
.IX Header "Author"
T.v.Dein <tom \s-1AT\s0 vondein \s-1DOT\s0 org>

View File

@@ -15,6 +15,7 @@ SYNOPSYS
--config -c <file> Use config file <file> (default: ~/.kleingebaeck).
--manual -m Show manual.
--help -h Show usage.
--version -V Show program version.
DESCRIPTION
This tool can be used to backup ads on the german ad page
@@ -89,7 +90,20 @@ LIMITATIONS
in the future.
LICENSE
Licensed under the GNU GENERAL PUBLIC LICENSE version 3.
Copyright 2023-2024 Thomas von Dein
This program is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation, either version 3 of the License, or (at your
option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.
You should have received a copy of the GNU General Public License along
with this program. If not, see <http://www.gnu.org/licenses/>.
Author
T.v.Dein <tom AT vondein DOT org>

View File

@@ -96,7 +96,20 @@ change in the future.
=head1 LICENSE
Licensed under the GNU GENERAL PUBLIC LICENSE version 3.
Copyright 2023-2024 Thomas von Dein
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see L<http://www.gnu.org/licenses/>.
=head1 Author

21
main.go
View File

@@ -20,6 +20,7 @@ package main
import (
"errors"
"fmt"
"io"
"log/slog"
"os"
"runtime/debug"
@@ -30,10 +31,10 @@ import (
const LevelNotice = slog.Level(2)
func main() {
os.Exit(Main())
os.Exit(Main(os.Stdout))
}
func Main() int {
func Main(w io.Writer) int {
logLevel := &slog.LevelVar{}
opts := &tint.Options{
Level: logLevel,
@@ -49,22 +50,22 @@ func Main() int {
}
logLevel.Set(LevelNotice)
var handler slog.Handler = tint.NewHandler(os.Stdout, opts)
handler := tint.NewHandler(w, opts)
logger := slog.New(handler)
slog.SetDefault(logger)
conf, err := InitConfig()
conf, err := InitConfig(w)
if err != nil {
return Die(err)
}
if conf.Showversion {
fmt.Printf("This is kleingebaeck version %s\n", VERSION)
fmt.Fprintf(w, "This is kleingebaeck version %s\n", VERSION)
return 0
}
if conf.Showhelp {
fmt.Println(Usage)
fmt.Fprintln(w, Usage)
return 0
}
@@ -90,7 +91,7 @@ func Main() int {
}
logLevel.Set(slog.LevelDebug)
var handler slog.Handler = tint.NewHandler(os.Stdout, opts)
handler := tint.NewHandler(w, opts)
debuglogger := slog.New(handler).With(
slog.Group("program_info",
slog.Int("pid", os.Getpid()),
@@ -100,6 +101,8 @@ func Main() int {
slog.SetDefault(debuglogger)
}
// defaultlogger := log.Default()
// defaultlogger.SetOutput(w)
slog.Debug("config", "conf", conf)
// prepare output dir
@@ -131,10 +134,10 @@ func Main() int {
if conf.StatsCountAds == 1 {
adstr = "ad"
}
fmt.Printf("Successfully downloaded %d %s with %d images to %s.\n",
fmt.Fprintf(w, "Successfully downloaded %d %s with %d images to %s.\n",
conf.StatsCountAds, adstr, conf.StatsCountImages, conf.Outdir)
} else {
fmt.Printf("No ads found.")
fmt.Fprintf(w, "No ads found.")
}
return 0

534
main_test.go Normal file
View File

@@ -0,0 +1,534 @@
/*
Copyright © 2023-2024 Thomas von Dein
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
package main
import (
"bytes"
"errors"
"fmt"
"os"
"strings"
"testing"
tpl "text/template"
"github.com/jarcoal/httpmock"
)
// the ad list, aka:
// https://www.kleinanzeigen.de/s-bestandsliste.html?userId=XXXXXX
// Note, that this HTML code is reduced to the max, so that it only
// contains the stuff required to satisfy goquery
const LISTTPL string = `<!DOCTYPE html>
<html lang="de" >
<head>
<title>Ads</title>
</head>
<body>
{{ range . }}
<h2 class="text-module-begin">
<a class="ellipsis"
href="/s-anzeige/{{ .Slug }}/{{ .Id }}">{{ .Title }}</a>
</h2>
{{ end }}
</body>
</html>
`
// an actual ad listing, aka:
// https://www.kleinanzeigen.de/s-anzeige/ad-text-slug/1010101010
// Note, that this HTML code is reduced to the max, so that it only
// contains the stuff required to satisfy goquery
const ADTPL string = `DOCTYPE html>
<html lang="de">
<head>
<title>Ad Listing</title>
</head>
<body>
<div class="l-container-row">
<div id="vap-brdcrmb" class="breadcrump">
<a class="breadcrump-link" itemprop="url" href="/" title="Kleinanzeigen ">
<span itemprop="title">Kleinanzeigen </span>
</a>
<a class="breadcrump-link" itemprop="url" href="/egal">
<span itemprop="title">{{ .Category }}</span></a>
</div>
</div>
{{ range $image := .Images }}
<div class="galleryimage-element" data-ix="3">
<img src="{{ $image }}"/>
</div>
{{ end }}
<h1 id="viewad-title" class="boxedarticle--title" itemprop="name" data-soldlabel="Verkauft">
{{ .Title }}</h1>
<div class="boxedarticle--flex--container">
<h2 class="boxedarticle--price" id="viewad-price">
{{ .Price }}</h2>
</div>
<div id="viewad-extra-info" class="boxedarticle--details--full">
<div><i class="icon icon-small icon-calendar-gray-simple"></i><span>{{ .Created }}</span></div>
</div>
<div class="splitlinebox l-container-row" id="viewad-details">
<ul class="addetailslist">
<li class="addetailslist--detail">
Zustand<span class="addetailslist--detail--value" >
{{ .Condition }}</span>
</li>
</ul>
</div>
<div class="l-container last-paragraph-no-margin-bottom">
<p id="viewad-description-text" class="text-force-linebreak " itemprop="description">
{{ .Text }}
</p>
</div>
</body>
</html>
`
const EMPTYPAGE string = `DOCTYPE html>
<html lang="de">
<head></head>
<body></body>
</html>
`
const (
EMPTYURI string = `https://www.kleinanzeigen.de/s-anzeige/empty/1`
INVALIDPATHURI string = `https://www.kleinanzeigen.de/anzeige/name/1`
INVALID404URI string = `https://www.kleinanzeigen.de/anzeige/name/1/foo/bar`
INVALIDURI string = `https://foo.bar/weird/things`
)
var base = "kleingebaeck -c t/config-empty.conf"
type Tests struct {
name string
args string
expect string
exitcode int
}
var tests = []Tests{
{
name: "version",
args: base + " -V",
expect: "This is",
exitcode: 0,
},
{
name: "help",
args: base + " -h",
expect: "Usage:",
exitcode: 0,
},
{
name: "debug",
args: base + " -d",
expect: "program_info",
exitcode: 1,
},
{
name: "no-args-no-user",
args: base,
expect: "invalid or no user id",
exitcode: 1,
},
{
name: "download-single-ad",
args: base + " -o t/out https://www.kleinanzeigen.de/s-anzeige/first-ad/1",
expect: "Successfully downloaded 1 ad with 2 images to t/out",
exitcode: 0,
},
{
name: "download-single-ad-verbose",
args: base + " -o t/out https://www.kleinanzeigen.de/s-anzeige/first-ad/1 -v",
expect: "wrote ad listing",
exitcode: 0,
},
{
name: "download-single-ad-debug",
args: base + " -o t/out https://www.kleinanzeigen.de/s-anzeige/first-ad/1 -d",
expect: "extracted ad listing program_info.pid=",
exitcode: 0,
},
{
name: "download-all-ads",
args: base + " -o t/out -u 1",
expect: "Successfully downloaded 6 ads with 12 images to t/out",
exitcode: 0,
},
{
name: "download-all-ads-using-config",
args: "kleingebaeck -c t/fullconfig.conf",
expect: "Successfully downloaded 6 ads with 12 images to t/out",
exitcode: 0,
},
}
var invalidtests = []Tests{
{
name: "empty-ad",
args: base + " " + EMPTYURI,
expect: "could not extract ad data from page, got empty struct",
exitcode: 1,
},
{
name: "invalid-ad",
args: base + " " + INVALIDURI,
expect: "invalid uri",
exitcode: 1,
},
{
name: "invalid-path",
args: base + " " + INVALIDPATHURI,
expect: "could not extract ad data from page, got empty struct",
exitcode: 1,
},
{
name: "404",
args: base + " " + INVALID404URI,
expect: "could not get page via HTTP",
exitcode: 1,
},
{
name: "outdir-no-exists",
args: base + " -o t/foo/bar/out https://www.kleinanzeigen.de/s-anzeige/first-ad/1 -v",
expect: "Failure",
exitcode: 1,
},
{
name: "wrong-flag",
args: base + " -X",
expect: "unknown shorthand flag: 'X' in -X",
exitcode: 1,
},
{
name: "no-config",
args: "kleingebaeck -c t/invalid.conf",
expect: "error loading config file",
exitcode: 1,
},
}
type AdConfig struct {
Title string
Slug string
Id string
Price string
Category string
Condition string
Created string
Text string
Images []string // files in ./t/
}
var adsrc = []AdConfig{
{
Title: "First Ad",
Id: "1", Price: "5€",
Category: "Klimbim",
Text: "Thing to sale",
Slug: "first-ad",
Condition: "works",
Created: "Yesterday",
Images: []string{"t/1.jpg", "t/2.jpg"},
},
{
Title: "Secnd Ad",
Id: "2", Price: "5€",
Category: "Kram",
Text: "Thing to sale",
Slug: "second-ad",
Condition: "works",
Created: "Yesterday",
Images: []string{"t/1.jpg", "t/2.jpg"},
},
{
Title: "Third Ad",
Id: "3",
Price: "5€",
Category: "Kuddelmuddel",
Text: "Thing to sale",
Slug: "third-ad",
Condition: "works",
Created: "Yesterday",
Images: []string{"t/1.jpg", "t/2.jpg"},
},
{
Title: "Forth Ad",
Id: "4",
Price: "5€",
Category: "Krempel",
Text: "Thing to sale",
Slug: "fourth-ad",
Condition: "works",
Created: "Yesterday",
Images: []string{"t/1.jpg", "t/2.jpg"},
},
{
Title: "Fifth Ad",
Id: "5",
Price: "5€",
Category: "Kladderadatsch",
Text: "Thing to sale",
Slug: "fifth-ad",
Condition: "works",
Created: "Yesterday",
Images: []string{"t/1.jpg", "t/2.jpg"},
},
{
Title: "Sixth Ad",
Id: "6",
Price: "5€",
Category: "Klunker",
Text: "Thing to sale",
Slug: "sixth-ad",
Condition: "works",
Created: "Yesterday",
Images: []string{"t/1.jpg", "t/2.jpg"},
},
}
// An Adsource is used to construct a httpmock responder for a
// particular url. So, the code (scrape.go) scrapes
// https://kleinanzeigen.de, but in reality httpmock captures the
// request and responds with our mock data
type Adsource struct {
uri string
content string
status int
}
// Render a HTML template for an adlisting or an ad
func GetTemplate(l []AdConfig, a AdConfig, htmltemplate string) string {
tmpl, err := tpl.New("template").Parse(htmltemplate)
if err != nil {
panic(err)
}
var out bytes.Buffer
if len(a.Id) == 0 {
err = tmpl.Execute(&out, l)
} else {
err = tmpl.Execute(&out, a)
}
if err != nil {
panic(err)
}
return out.String()
}
// Initialize the valid sources for the httpmock responder
func InitValidSources() []Adsource {
// valid ad listing page 1
list1 := []AdConfig{
adsrc[0], adsrc[1], adsrc[2],
}
// valid ad listing page 2
list2 := []AdConfig{
adsrc[3], adsrc[4], adsrc[5],
}
// valid ad listing page 3, which is empty
list3 := []AdConfig{}
// used to signal GetTemplate() to render a listing
empty := AdConfig{}
// prepare urls for the listing pages
ads := []Adsource{
{
uri: fmt.Sprintf("%s%s?userId=1", Baseuri, Listuri),
content: GetTemplate(list1, empty, LISTTPL),
},
{
uri: fmt.Sprintf("%s%s?userId=1&pageNum=2", Baseuri, Listuri),
content: GetTemplate(list2, empty, LISTTPL),
},
{
uri: fmt.Sprintf("%s%s?userId=1&pageNum=3", Baseuri, Listuri),
content: GetTemplate(list3, empty, LISTTPL),
},
}
// prepare urls for the ads
for _, ad := range adsrc {
ads = append(ads, Adsource{
uri: fmt.Sprintf("%s/s-anzeige/%s/%s", Baseuri, ad.Slug, ad.Id),
content: GetTemplate(nil, ad, ADTPL),
})
//panic(GetTemplate(nil, ad, ADTPL))
}
return ads
}
func InitInvalidSources() []Adsource {
empty := AdConfig{}
ads := []Adsource{
{
// valid ad page but without content
uri: fmt.Sprintf("%s/s-anzeige/empty/1", Baseuri),
content: GetTemplate(nil, empty, EMPTYPAGE),
},
{
// some random foreign webpage
uri: INVALIDURI,
content: GetTemplate(nil, empty, "<html>foo</html>"),
},
{
// some invalid page path
uri: fmt.Sprintf("%s/anzeige/name/1", Baseuri),
content: GetTemplate(nil, empty, "<html></html>"),
},
{
// some none-ad page
uri: fmt.Sprintf("%s/anzeige/name/1/foo/bar", Baseuri),
content: GetTemplate(nil, empty, "<html>HTTP 404: /eine-anzeige/ does not exist!</html>"),
status: 404,
},
}
return ads
}
// load a test image from disk
func GetImage(path string) []byte {
dat, err := os.ReadFile(path)
if err != nil {
panic(err)
}
return dat
}
// setup httpmock
func SetIntercept(ads []Adsource) {
for _, ad := range ads {
if ad.status == 0 {
ad.status = 200
}
httpmock.RegisterResponder("GET", ad.uri,
httpmock.NewStringResponder(ad.status, ad.content))
}
// we just use 2 images, put this here
for _, image := range []string{"t/1.jpg", "t/2.jpg"} {
httpmock.RegisterResponder("GET", image, httpmock.NewBytesResponder(200, GetImage(image)))
}
}
func VerifyAd(ad AdConfig) error {
body := ad.Title + ad.Price + ad.Id + "Kleinanzeigen => " + ad.Category + ad.Condition + ad.Created
// prepare ad dir name using DefaultAdNameTemplate
c := Config{Adnametemplate: DefaultAdNameTemplate}
adstruct := Ad{Slug: ad.Slug, Id: ad.Id}
addir, err := AdDirName(&c, &adstruct)
if err != nil {
return err
}
file := fmt.Sprintf("t/out/%s/Adlisting.txt", addir)
content, err := os.ReadFile(file)
if err != nil {
return err
}
if body != strings.TrimSpace(string(content)) {
msg := fmt.Sprintf("ad content doesn't match.\nExpect: %s\n Got: %s\n", body, content)
return errors.New(msg)
}
return nil
}
func TestMain(t *testing.T) {
oldargs := os.Args
defer func() { os.Args = oldargs }()
httpmock.Activate()
defer httpmock.DeactivateAndReset()
// prepare httpmock responders
SetIntercept(InitValidSources())
// run commandline tests
for _, tt := range tests {
var buf bytes.Buffer
os.Args = strings.Split(tt.args, " ")
ret := Main(&buf)
if ret != tt.exitcode {
t.Errorf("%s with cmd <%s> did not exit with %d but %d",
tt.name, tt.args, tt.exitcode, ret)
}
if !strings.Contains(buf.String(), tt.expect) {
t.Errorf("%s with cmd <%s> output did not match.\nExpect: %s\n Got: %s\n",
tt.name, tt.args, tt.expect, buf.String())
}
}
// verify if downloaded ads match
for _, ad := range adsrc {
if err := VerifyAd(ad); err != nil {
t.Errorf(err.Error())
}
}
}
func TestMainInvalids(t *testing.T) {
oldargs := os.Args
defer func() { os.Args = oldargs }()
httpmock.Activate()
defer httpmock.DeactivateAndReset()
// prepare httpmock responders
SetIntercept(InitInvalidSources())
// run commandline tests
for _, tt := range invalidtests {
var buf bytes.Buffer
os.Args = strings.Split(tt.args, " ")
ret := Main(&buf)
if ret != tt.exitcode {
t.Errorf("%s with cmd <%s> did not exit with %d but %d",
tt.name, tt.args, tt.exitcode, ret)
}
if !strings.Contains(buf.String(), tt.expect) {
t.Errorf("%s with cmd <%s> output did not match.\nExpect: %s\n Got: %s\n",
tt.name, tt.args, tt.expect, buf.String())
}
}
}

View File

@@ -1,5 +1,5 @@
/*
Copyright © 2023 Thomas von Dein
Copyright © 2023-2024 Thomas von Dein
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -30,33 +30,6 @@ import (
"golang.org/x/sync/errgroup"
)
type Index struct {
Links []string `goquery:".text-module-begin a,[href]"`
}
type Ad struct {
Title string `goquery:"h1"`
Slug string
Id string
Condition string
Category string
Price string `goquery:"h2#viewad-price"`
Created string `goquery:"#viewad-extra-info,text"`
Text string `goquery:"p#viewad-description-text,html"`
Images []string `goquery:".galleryimage-element img,[src]"`
Meta []string `goquery:".addetailslist--detail--value,text"`
}
func (ad *Ad) LogValue() slog.Value {
return slog.GroupValue(
slog.String("title", ad.Title),
slog.String("price", ad.Price),
slog.String("id", ad.Id),
slog.Int("imagecount", len(ad.Images)),
slog.Int("bodysize", len(ad.Text)),
)
}
// fetch some web page content
func Get(uri string, client *http.Client) (io.ReadCloser, error) {
req, err := http.NewRequest("GET", uri, nil)
@@ -74,6 +47,10 @@ func Get(uri string, client *http.Client) (io.ReadCloser, error) {
slog.Debug("response", "code", res.StatusCode, "status",
res.Status, "size", res.ContentLength)
if res.StatusCode != 200 {
return nil, errors.New("could not get page via HTTP")
}
return res.Body, nil
}
@@ -140,7 +117,7 @@ func Scrape(c *Config, uri string) error {
// extract slug and id from uri
uriparts := strings.Split(uri, "/")
if len(uriparts) < 6 {
return errors.New("invalid uri")
return errors.New("invalid uri: " + uri)
}
ad.Slug = uriparts[4]
ad.Id = uriparts[5]
@@ -158,31 +135,37 @@ func Scrape(c *Config, uri string) error {
if err != nil {
return err
}
if len(ad.Meta) == 2 {
ad.Category = ad.Meta[0]
ad.Condition = ad.Meta[1]
if len(ad.CategoryTree) > 0 {
ad.Category = strings.Join(ad.CategoryTree, " => ")
}
if ad.Incomplete() {
slog.Debug("got ad", "ad", ad)
return errors.New("could not extract ad data from page, got empty struct")
}
slog.Debug("extracted ad listing", "ad", ad)
// write listing
err = WriteAd(c.Outdir, ad, c.Template)
addir, err := WriteAd(c, ad)
if err != nil {
return err
}
c.IncrAds()
return ScrapeImages(c, ad)
return ScrapeImages(c, ad, addir)
}
func ScrapeImages(c *Config, ad *Ad) error {
func ScrapeImages(c *Config, ad *Ad, addir string) error {
// fetch images
img := 1
g := new(errgroup.Group)
for _, imguri := range ad.Images {
imguri := imguri
file := filepath.Join(c.Outdir, ad.Slug, fmt.Sprintf("%d.jpg", img))
file := filepath.Join(c.Outdir, addir, fmt.Sprintf("%d.jpg", img))
g.Go(func() error {
err := Getimage(imguri, file)
if err != nil {
@@ -214,7 +197,7 @@ func Getimage(uri, fileName string) error {
defer response.Body.Close()
if response.StatusCode != 200 {
return errors.New("received non 200 response code")
return errors.New("could not get image via HTTP")
}
err = WriteImage(fileName, response.Body)

View File

@@ -1,5 +1,5 @@
/*
Copyright © 2023 Thomas von Dein
Copyright © 2023-2024 Thomas von Dein
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -18,6 +18,7 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
package main
import (
"bytes"
"io"
"log/slog"
"os"
@@ -27,20 +28,42 @@ import (
tpl "text/template"
)
func WriteAd(dir string, ad *Ad, template string) error {
// prepare output dir
dir = filepath.Join(dir, ad.Slug)
err := Mkdir(dir)
func AdDirName(c *Config, ad *Ad) (string, error) {
tmpl, err := tpl.New("adname").Parse(c.Adnametemplate)
if err != nil {
return err
return "", err
}
buf := bytes.Buffer{}
err = tmpl.Execute(&buf, ad)
if err != nil {
return "", err
}
return buf.String(), nil
}
func WriteAd(c *Config, ad *Ad) (string, error) {
// prepare ad dir name
addir, err := AdDirName(c, ad)
if err != nil {
return "", err
}
// prepare output dir
dir := filepath.Join(c.Outdir, addir)
err = Mkdir(dir)
if err != nil {
return "", err
}
// write ad file
listingfile := filepath.Join(dir, "Adlisting.txt")
f, err := os.Create(listingfile)
if err != nil {
return err
return "", err
}
defer f.Close()
if runtime.GOOS == "windows" {
ad.Text = strings.ReplaceAll(ad.Text, "<br/>", "\r\n")
@@ -48,18 +71,19 @@ func WriteAd(dir string, ad *Ad, template string) error {
ad.Text = strings.ReplaceAll(ad.Text, "<br/>", "\n")
}
tmpl, err := tpl.New("adlisting").Parse(template)
tmpl, err := tpl.New("adlisting").Parse(c.Template)
if err != nil {
return err
return "", err
}
err = tmpl.Execute(f, ad)
if err != nil {
return err
return "", err
}
slog.Info("wrote ad listing", "listingfile", listingfile)
return nil
return addir, nil
}
func WriteImage(filename string, reader io.ReadCloser) error {

BIN
t/1.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1001 B

BIN
t/2.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1002 B

6
t/config-empty.conf Normal file
View File

@@ -0,0 +1,6 @@
# empty config for Main() unit tests to force unit tests NOT to use an
# eventually existing ~/.kleingebaeck!
template="""
{{.Title}}{{.Price}}{{.Id}}{{.Category}}{{.Condition}}{{.Created}}
"""

6
t/fullconfig.conf Normal file
View File

@@ -0,0 +1,6 @@
user = 1
loglevel = "verbose"
outdir = "t/out"
template="""
{{.Title}}{{.Price}}{{.Id}}{{.Category}}{{.Condition}}{{.Created}}
"""

1
t/invalid.conf Normal file
View File

@@ -0,0 +1 @@
user = "