32 Commits

Author SHA1 Message Date
T.v.Dein
15b2e7f2a7 update go, dependencies, catch Fprint and Close errors (#138)
Co-authored-by: Thomas von Dein <tom@vondein.org>
2025-05-04 12:05:59 +02:00
T.v.Dein
4030d04b06 Add year,month,day support to Adnametemplate (#123)
* add year, month and day to adnametemplate as well
2025-02-27 17:58:05 +01:00
T.v.Dein
46be48af38 Generic attributes (#120)
* fix #117: use a generic attribute parser, still support fixed attrs
2025-02-10 18:20:54 +01:00
T.v.Dein
09948a6b39 add color detail as well (#119)
Co-authored-by: Thomas von Dein <tom@vondein.org>
2025-02-06 20:13:08 +01:00
T.v.Dein
bc01391872 Fix ad condition parsing (#118)
* fix #117: use details slice and pre-set to properly extract condition
* also added the type part of the detail content (original de: "Art")

---------

Co-authored-by: Thomas von Dein <tom@vondein.org>
2025-02-06 13:48:20 +01:00
fbd9a5a621 make clear that this is not the complete filename 2025-01-06 18:42:19 +01:00
2239a83f76 properly check image format for storing and distance hashing 2025-01-06 18:42:19 +01:00
2288806105 fix #77: use processed ad dir for duplicate checking, not slug 2024-02-10 15:15:43 +01:00
612ed2aa79 fix #74: warn if about to write to already visited ad, overwrite if -f 2024-02-10 14:44:09 +01:00
ed78731b3c check seek error 2024-01-27 17:34:44 +01:00
a84f0e1436 get rid of duplicate bytes.Buffer, use bytes.Reader instead, #39 2024-01-27 17:34:44 +01:00
T.v.Dein
14f8c3fd43 Fix/linter (#66)
* added lint targets
* fix linter errors
* enhance error handling
* !!BREAKING!! rename Id to ID in tpls
2024-01-25 19:04:15 +01:00
8df3ebfa6d add throttling to image download 2024-01-24 19:22:31 +01:00
71c528114f fix linter error 2024-01-22 18:43:44 +01:00
e2afc1350b added -f to override d-hash, better debug and error handling 2024-01-22 18:43:44 +01:00
e971070f9f added image diff hash distance caching to not overwrite similar images 2024-01-22 18:43:44 +01:00
T.v.Dein
8771ec1108 added support to calculate and store the ad expire date (#43) 2024-01-19 18:38:35 +01:00
T.v.Dein
42a958fc4c refactored out http fetching code into Fetcher{}/fetch.go 2024-01-19 18:38:35 +01:00
T.v.Dein
5fa46ff106 Add HTTP retries and the possibility to ignore image download errors (#33)
added HTTP retry and --ignoreerrors which ignores image download errors, fix #30
2024-01-19 18:38:35 +01:00
T.v.Dein
cca3211023 Enhancement/http (#32)
* added HTTP debug logging using `-d` or `DEBUGHTTP=1` (headers only)
2024-01-19 18:38:35 +01:00
0fd9b519d1 fixed changes on kleinanzeigen.de:
- Meta did not contain condition and category together anymore, they
removed  the category. Therefore fetching (that is, validation)
failed.
- Now we extract the condition and category directly.
- On top, category now includes the whole category tree.
- unit tests had to be tweaked for this measure.
2024-01-19 18:38:35 +01:00
5abbab9527 added template for ad directory, by default include id now 2024-01-19 18:38:35 +01:00
0baaf6f38b better error message on 404 2024-01-19 18:38:35 +01:00
1ff5c240c8 put ad code into separate file, enhance error checking 2024-01-19 18:38:35 +01:00
T.v.Dein
f893f9c3d7 Test/add mock tests (#24)
* add scrape unit test using httpmock lib
2024-01-19 18:38:35 +01:00
T.v.Dein
b8977df986 Bugfixes (#13)
* several fixes:

- fix #9 + #10: switched to koanf module and dropped support for HCL
- fix #11: disabling colors on windows
- fix #12: fixed race condition in go routine call inside for loop,
  images had been downloaded multiple times
- remove hcl support and use toml format (same thing, better parser)
- update documentation and example config on TOML format of config file
- use Config as arg instead of singular args
- use x/errgroup instead of sync.Waitgroup inside image download loop

---------

Co-authored-by: Thomas von Dein <tom@vondein.org>
2024-01-19 18:38:35 +01:00
4d4577c9f8 fix #4, use filepath.Join to create portable path's 2024-01-19 18:38:27 +01:00
f932d7be83 re-orgainzied code a little, using go templates instead format string 2023-12-17 17:32:05 +01:00
e904ed6687 added custom template support, added more ad data, use concurrency 2023-12-16 20:32:10 +01:00
1b55d887bc enhancements:
- english README (german version will be put to the homepage)
- better commandline options
- enhanced logging capabilities and error handling
- config file support
- support to backup one or more singular ads
- add id to adlisting
- added manual page
- fixed config file reading
- fixed typo
2023-12-15 17:19:44 +01:00
5bd49db9ba scrape all ads 2023-12-14 19:02:32 +01:00
12329f6ae0 initial commit 2023-12-14 19:00:04 +01:00