Commit Graph

21 Commits

Author SHA1 Message Date
T.v.Dein
14f8c3fd43 Fix/linter (#66)
* added lint targets
* fix linter errors
* enhance error handling
* !!BREAKING!! rename Id to ID in tpls
2024-01-25 19:04:15 +01:00
8df3ebfa6d add throttling to image download 2024-01-24 19:22:31 +01:00
71c528114f fix linter error 2024-01-22 18:43:44 +01:00
e2afc1350b added -f to override d-hash, better debug and error handling 2024-01-22 18:43:44 +01:00
e971070f9f added image diff hash distance caching to not overwrite similar images 2024-01-22 18:43:44 +01:00
T.v.Dein
8771ec1108 added support to calculate and store the ad expire date (#43) 2024-01-19 18:38:35 +01:00
T.v.Dein
42a958fc4c refactored out http fetching code into Fetcher{}/fetch.go 2024-01-19 18:38:35 +01:00
T.v.Dein
5fa46ff106 Add HTTP retries and the possibility to ignore image download errors (#33)
added HTTP retry and --ignoreerrors which ignores image download errors, fix #30
2024-01-19 18:38:35 +01:00
T.v.Dein
cca3211023 Enhancement/http (#32)
* added HTTP debug logging using `-d` or `DEBUGHTTP=1` (headers only)
2024-01-19 18:38:35 +01:00
0fd9b519d1 fixed changes on kleinanzeigen.de:
- Meta did not contain condition and category together anymore, they
removed  the category. Therefore fetching (that is, validation)
failed.
- Now we extract the condition and category directly.
- On top, category now includes the whole category tree.
- unit tests had to be tweaked for this measure.
2024-01-19 18:38:35 +01:00
5abbab9527 added template for ad directory, by default include id now 2024-01-19 18:38:35 +01:00
0baaf6f38b better error message on 404 2024-01-19 18:38:35 +01:00
1ff5c240c8 put ad code into separate file, enhance error checking 2024-01-19 18:38:35 +01:00
T.v.Dein
f893f9c3d7 Test/add mock tests (#24)
* add scrape unit test using httpmock lib
2024-01-19 18:38:35 +01:00
T.v.Dein
b8977df986 Bugfixes (#13)
* several fixes:

- fix #9 + #10: switched to koanf module and dropped support for HCL
- fix #11: disabling colors on windows
- fix #12: fixed race condition in go routine call inside for loop,
  images had been downloaded multiple times
- remove hcl support and use toml format (same thing, better parser)
- update documentation and example config on TOML format of config file
- use Config as arg instead of singular args
- use x/errgroup instead of sync.Waitgroup inside image download loop

---------

Co-authored-by: Thomas von Dein <tom@vondein.org>
2024-01-19 18:38:35 +01:00
4d4577c9f8 fix #4, use filepath.Join to create portable path's 2024-01-19 18:38:27 +01:00
f932d7be83 re-orgainzied code a little, using go templates instead format string 2023-12-17 17:32:05 +01:00
e904ed6687 added custom template support, added more ad data, use concurrency 2023-12-16 20:32:10 +01:00
1b55d887bc enhancements:
- english README (german version will be put to the homepage)
- better commandline options
- enhanced logging capabilities and error handling
- config file support
- support to backup one or more singular ads
- add id to adlisting
- added manual page
- fixed config file reading
- fixed typo
2023-12-15 17:19:44 +01:00
5bd49db9ba scrape all ads 2023-12-14 19:02:32 +01:00
12329f6ae0 initial commit 2023-12-14 19:00:04 +01:00