3245438564
update yadu to v0.1.1, fix #46
2024-01-22 18:43:44 +01:00
71c528114f
fix linter error
2024-01-22 18:43:44 +01:00
e2afc1350b
added -f to override d-hash, better debug and error handling
2024-01-22 18:43:44 +01:00
e971070f9f
added image diff hash distance caching to not overwrite similar images
2024-01-22 18:43:44 +01:00
T.v.Dein
8771ec1108
added support to calculate and store the ad expire date ( #43 )
2024-01-19 18:38:35 +01:00
T.v.Dein
1896209b96
use new yadu log handler, +tests, +upd modules, +version
2024-01-19 18:38:35 +01:00
T.v.Dein
3c93c9fce0
added docker image support
...
* added environment variable support
* added docker instructions
* added .env hint
2024-01-19 18:38:35 +01:00
T.v.Dein
42a958fc4c
refactored out http fetching code into Fetcher{}/fetch.go
2024-01-19 18:38:35 +01:00
T.v.Dein
5fa46ff106
Add HTTP retries and the possibility to ignore image download errors ( #33 )
...
added HTTP retry and --ignoreerrors which ignores image download errors, fix #30
2024-01-19 18:38:35 +01:00
T.v.Dein
cca3211023
Enhancement/http ( #32 )
...
* added HTTP debug logging using `-d` or `DEBUGHTTP=1` (headers only)
2024-01-19 18:38:35 +01:00
T.v.Dein
dce7604afb
fix #30 : revert default adnamedir to just use the slug as before ( #31 )
2024-01-19 18:38:35 +01:00
0fd9b519d1
fixed changes on kleinanzeigen.de:
...
- Meta did not contain condition and category together anymore, they
removed the category. Therefore fetching (that is, validation)
failed.
- Now we extract the condition and category directly.
- On top, category now includes the whole category tree.
- unit tests had to be tweaked for this measure.
2024-01-19 18:38:35 +01:00
6b7f727449
fixed utf8
2024-01-19 18:38:35 +01:00
5abbab9527
added template for ad directory, by default include id now
2024-01-19 18:38:35 +01:00
T.v.Dein
e03c7debb6
remove duplicate license badge ( #28 )
...
* remove duplicate license badge
* fix badges
2024-01-19 18:38:35 +01:00
1d2483d18f
portable error check
2024-01-19 18:38:35 +01:00
b17f4f0f3e
also added coverage report+badge
2024-01-19 18:38:35 +01:00
4a91167871
put all tests into main_test.go, more failure mode tests and verify
2024-01-19 18:38:35 +01:00
0baaf6f38b
better error message on 404
2024-01-19 18:38:35 +01:00
42182bb6c9
add commandline main() test units
2024-01-19 18:38:35 +01:00
8455c193eb
pass a io.Writer to loggers and outputs so we can test the cmdline
2024-01-19 18:38:35 +01:00
d1faa10a52
added more invalid tests
2024-01-19 18:38:35 +01:00
e28137bf9b
upd httpmock+deps
2024-01-19 18:38:35 +01:00
1ff5c240c8
put ad code into separate file, enhance error checking
2024-01-19 18:38:35 +01:00
T.v.Dein
f893f9c3d7
Test/add mock tests ( #24 )
...
* add scrape unit test using httpmock lib
2024-01-19 18:38:35 +01:00
T.v.Dein
c4e88d98f2
fix linter errors ( #23 )
2024-01-19 18:38:35 +01:00
T.v.Dein
0cca387982
add ci pipeline ( #22 )
...
Co-authored-by: Thomas von Dein <tom@izb.net >
2024-01-19 18:38:35 +01:00
T.v.Dein
9e619fb3c5
Doc/add prior art ( #21 )
...
* add mor prior art
2024-01-19 18:38:35 +01:00
T.v.Dein
0fdfed2929
added windows screenshots ( #20 )
...
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
T.v.Dein
73c09ec38b
Revert "Fix/newline windows ( #18 )" ( #19 )
...
This reverts commit eaf4db6cef
2024-01-19 18:38:35 +01:00
T.v.Dein
f901af4f0c
Fix/newline windows ( #18 )
...
* fix #17 : use fmt.Println() after stats
* bump version
2024-01-19 18:38:35 +01:00
T.v.Dein
2a8f53ca98
added screenshots and a section about prior work ( #16 )
2024-01-19 18:38:35 +01:00
T.v.Dein
4a95cb1f5e
add doc link ( #15 )
...
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
T.v.Dein
482612f889
fix invalid mod load ( #14 )
...
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
T.v.Dein
b8977df986
Bugfixes ( #13 )
...
* several fixes:
- fix #9 + #10 : switched to koanf module and dropped support for HCL
- fix #11 : disabling colors on windows
- fix #12 : fixed race condition in go routine call inside for loop,
images had been downloaded multiple times
- remove hcl support and use toml format (same thing, better parser)
- update documentation and example config on TOML format of config file
- use Config as arg instead of singular args
- use x/errgroup instead of sync.Waitgroup inside image download loop
---------
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
T.v.Dein
ae5e3daea3
Dev ( #8 )
...
* fixed conf parsing: variables can now be omitted from the config
* fix newlines: use CRLF on windows
* bump version
---------
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
T.v.Dein
1c6d832b20
added proper install instructions ( #7 )
...
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
52b39d91a3
fix version finding + bump version
2024-01-19 18:38:27 +01:00
3748cd35e5
fix #5 : add exe extension to built windows binaries
2024-01-19 18:38:27 +01:00
4d4577c9f8
fix #4 , use filepath.Join to create portable path's
2024-01-19 18:38:27 +01:00
f932d7be83
re-orgainzied code a little, using go templates instead format string
2023-12-17 17:32:05 +01:00
e904ed6687
added custom template support, added more ad data, use concurrency
2023-12-16 20:32:10 +01:00
df6baadc85
better sample config
2023-12-16 00:01:29 +01:00
314315a1c6
fix pod entities => markdown
2023-12-15 18:29:42 +01:00
2e83e68f20
fix logo
2023-12-15 18:02:34 +01:00
b5e51b43c9
add logo
2023-12-15 18:00:41 +01:00
1b55d887bc
enhancements:
...
- english README (german version will be put to the homepage)
- better commandline options
- enhanced logging capabilities and error handling
- config file support
- support to backup one or more singular ads
- add id to adlisting
- added manual page
- fixed config file reading
- fixed typo
2023-12-15 17:19:44 +01:00
c2f378be05
fix link
2023-12-14 19:06:40 +01:00
5b47128d0d
added name
2023-12-14 19:03:18 +01:00
5bd49db9ba
scrape all ads
2023-12-14 19:02:32 +01:00
